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April  7,  1997 

MEMORANDUM  FOR  DEPUTY  UNDER  SECRETARY  OF  DEFENSE 

(ADVANCED  TECHNOLOGY) 

SUBJECT:  Audit  Report  on  Advanced  Concept  Technology  Demonstration 
(Report  No.  97-120) 


We  are  providing  this  final  audit  report  for  information  and  use.  We  considered 
management  comments  on  the  draft  of  this  report  in  preparing  the  final  report. 


Comments  on  the  draft  of  this  report  conformed  to  the  requirements  of  DoD 
Directive  7650.3  and  left  no  unresolved  issues.  Therefore,  no  additional  comments  are 
required. 


We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  on  the  audit 
should  be  directed  to  Mr.  Raymond  A.  Spencer,  Audit  Program  Director,  at 
(7031  604-9071  (DSN  664-9071)  or  Mr.  Thomas  S.  Bartoszek,  Audit  Project  Manager, 
at  (703)  604-9014  (DSN  664-9014).  See  Appendix  G  for  the  report  distribution.  The 
audit  team  members  are  listed  inside  the  back  cover. 


Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


Report  No.  97-120  April  7,  1997 

(Project  No.  6AB-0018) 

Advanced  Concept  Technology  Demonstration 


Executive  Summary 


Introduction.  DoD  began  the  Advanced  Concept  Technology  Demonstration  program 
in  1994  to  provide  enhanced  capabilities  to  the  Military  Departments  through  the 
integration  of  mature  and  cost-effective  technologies.  Three  themes  form  the  focus  of 
the  Advanced  Concept  Technology  Demonstration  program:  user  involvement  from  the 
inception  of  the  program,  mature  technology  focused  on  warfighting  needs,  and  a  plan 
to  execute  a  focused  demonstration  program  that  will  provide  the  user  a  sound  basis  for 
investment  decisions  before  commitment  to  a  formal  acquisition  program.  The  projects 
are  not  Defense  acquisition  programs  as  defined  in  DoD  Directive  5000.1,  "Defense 
Acquisition,"  and,  therefore,  do  not  follow  the  same  procedures  and  not  subject  to  the 
same  oversight  as  Defense  acquisition  programs. 

The  Deputy  Under  Secretary  of  Defense  (Advanced  Technology)  approved  22  projects, 
estimated  to  cost  about  $4  billion,  for  FYs  1995  and  1996.  The  Deputy  Under 
Secretary  provides  oversight,  support,  and  evaluation  of  the  approved  projects.  As  the 
program  and  its  processes  have  evolved,  the  Deputy  Under  Secretary  identified  and 
corrected  problems  with  the  process  through  the  integrated  product  team  process  and 
has  clarified  the  procedures. 

Audit  Objectives.  The  overall  audit  objective  was  to  evaluate  the  Advanced  Concept 
Technology  Demonstration  program.  In  this  respect,  we  reviewed  criteria  used  to 
select  current  and  pending  Advanced  Concept  Technology  Demonstration  efforts,  the 
process  for  determining  the  program's  effectiveness,  and  the  transition  of  the  program 
into  the  Defense  acquisition  cycle.  We  also  evaluated  the  adequacy  of  the  DoD 
Components'  management  controls  as  they  apply  to  the  audit  objectives. 

Audit  Results.  Based  on  our  interpretation  of  the  selection  criteria,  five  projects, 
valued  at  $2.3  billion,  of  nine  projects  that  we  selected  for  review  were  questionable 
choices  as  advanced  concept  technology  demonstration  projects.  As  a  result,  the 
Advanced  Concept  Technology  Demonstration  process  lacked  broad  support  within  the 
Military  Departments,  and  the  Military  Departments  were  unclear  on  what  type  of 
projects  would  make  viable  projects.  The  Deputy  Under  Secretary  of  Defense 
(Advanced  Technology)  has  issued  guidance  for  transitioning  advanced  concept 
technology  demonstration  programs  into  the  acquisition  process,  clarified  some  of  the 
criteria,  and  put  the  criteria  and  guidance  in  the  Defense  Acquisition  Deskbook  so  that 
all  parties  to  the  advanced  concept  technology  demonstration  have  access  to  the  criteria. 
However,  two  critical  criteria  for  selecting  advanced  concept  technology  demonstration 
programs,  mature  technology  and  critical  military  need,  are  still  undefined. 

Recommendations  in  this  report,  if  implemented,  will  improve  the  effectiveness  of  the 
selection  process  for  the  Advanced  Concept  Technology  Demonstration  program  and 
enhance  support  for  it.  See  Part  I  for  a  detailed  discussion  of  the  audit  results.  See 
Appendix  A  for  the  management  control  program  discussion. 
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Summary  of  Recommendations.  We  recommend  that  the  Deputy  Under  Secretary  of 
Defense  (Advanced  Technology)  develop  clear  and  assessable  selection  criteria  to 
include  the  definitions  of  mature  technology  and  critical  military  need,  include  those 
definitions  in  the  Defense  Acquisition  Deskbook,  and  reevaluate  the  FY  1997 
candidates  when  the  criteria  are  developed. 

Management  Comments.  The  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  did  not  agree  with  the  conclusion  that  some  projects  were  questionable 
choices  under  the  criteria.  However,  he  stated  that  the  fact  that  the  issue  was  raised 
indicated  a  need  to  clarify  the  criteria.  Therefore,  he  concurred  with  our 
recommendations.  Although  not  required  to  comment  on  the  draft  audit  report,  the 
Vice  Chairman  of  the  Joint  Chiefs  of  Staff  also  stated  that  he  endorses  the 
recommendations.  The  comments  are  in  Part  III  of  this  report.  Our  response  to  the 
Under  Secretary's  comments  is  in  Appendix  F.  We  will  be  pleased  to  participate  in 
any  group  established  to  implement  our  recommendations. 
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Part  I  -  Audit  Results 
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Audit  Results 


Audit  Background 


The  Deputy  Under  Secretary  of  Defense  (Advanced  Technology)  (DUSD[AT]) 
initiated  the  Advanced  Concept  Technology  Demonstration  (ACTD)  program  in 
early  1994  to  enhance  military  capability  within  a  short  timeframe  to  solve 
important  military  problems  that  the  user  identified.  The  ACTDs  require  user 
involvement  from  inception  to  evaluate  military  utility,  develop  operational 
concepts,  and  retain  low-cost  residual  operational  capabilities. 

As  the  program  and  its  processes  have  evolved,  DUSD(AT)  has  identified  and 
corrected  problems  with  the  process  through  the  integrated  product  team  process 
and  has  clarified  the  procedures.  The  DUSD(AT)  also  defines  guidelines, 
provides  oversight  support  through  an  oversight  committee,  provides  evaluations 
of  the  projects,  and  approves  projects  for  the  program. 

The  DUSD(AT)  approved  22  ACTDs,  valued  at  about  $4  billion,  for  FYs  1995 
and  1996  (Appendix  B). 


Audit  Objectives 


We  evaluated  Office  of  the  Secretary  of  Defense  and  DoD  Component  policies 
and  procedures  for  the  ACTD  program  to  ensure  that  appropriate  candidates 
were  selected  for  the  program,  that  the  programs  were  managed  effectively,  and 
that  their  transition  into  the  Defense  acquisition  cycle  was  adequate.  We  also 
evaluated  the  management  controls  as  they  applied  to  the  audit  objectives.  The 
audit  scope  and  methodology,  prior  audits  and  other  reviews,  review  of  the 
management  control  program,  and  organizations  and  individuals  visited  or 
contacted  are  in  Appendix  A. 

Also  included  in  Appendix  A  is  a  discussion  of  the  areas  not  requiring  further 
review  because  of  effective  policies  or  effective  management.  Those  areas  are 
the  effectiveness  of  the  ACTDs  and  the  transition  of  ACTDs  into  the  Defense 
acquisition  cycle. 
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The  Advanced  Concept  Technology 
Demonstration  Selection  Process 

Based  on  our  interpretation  of  the  selection  criteria,  five  projects,  valued 
at  $2.3  billion,  of  nine  projects  that  we  selected  for  review  and  that  the 
DUSD(AT)  approved,  were  questionable  ACTDs.  The  projects  were 
questionable  because  the  DUSD(AT)  had  not  clearly  defined  the 
selection  criteria  and,  therefore,  inconsistently  applied  the  criteria.  Four 
projects  do  not  have  mature  technology,  and  eight  projects  do  not  meet  a 
critical  military  need.  As  a  result,  the  ACTD  process  lacks  broad 
support  within  the  Military  Departments,  and  the  Military  Departments 
were  unclear  on  what  type  of  projects  would  make  viable  ACTDs.  Of 
the  22  projects  selected  for  FYs  1995  and  1996,  the  Military 
Departments  proposed  only  7  of  the  projects  for  the  ACTD  program.  In 
addition,  Congress  expressed  concern  about  the  use  of  the  ACTDs. 


ACTD  Policy 


DoD  Regulation.  DoD  Regulation  5000. 2-R,  "Mandatory  Procedures  for 
Major  Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated 
Information  Systems  (MAIS)  Acquisition  Programs,"  March  15,  1996,  part  2.7, 
"Advanced  Concept  Technology  Demonstration,"  states  that  ACTDs  are  a 
means  to  demonstrate  the  use  of  mature  technology  to  address  urgent  military 
needs  (During  the  audit,  DUSD[AT]  changed  the  terminology  from  "urgent"  to 
"critical.").  While  stating  that  the  ACTDs  are  not  an  acquisition  program 
themselves,  the  regulation  states  that  additional  buys  beyond  any  residual 
capability  of  the  project  would  become  an  acquisition  program. 

Office  of  the  Secretary  of  Defense  Guidance.  The  "ACTD  Initiation  and 
Approval  Process,"  July  1995,  published  by  DUSD(AT),  states  that  the  ACTDs 
initiation  and  approval  process  stresses  flexibility,  senior  management 
involvement,  and  judgment  instead  of  detailed  criteria  for  selecting  ACTDs. 
The  ACTD  process  begins  with  a  user  and  developer  teaming  to  propose 
projects  to  DUSD(AT).  If  DUSD(AT)  accepts  the  proposed  project  as  an 
ACTD  candidate,  then  both  the  Joint  Staff  and  DUSD(AT)  Breakfast  Club  (the 
Breakfast  Club)  assess  it  separately.  The  Breakfast  Club  is  an  advisory  group 
consisting  of  executives  from  DUSD(AT);  the  Director,  Defense  Research  and 
Engineering;  the  Joint  Staff;  and  the  DoD  Components'  Science  and 
Technology  and  Operations  groups.  After  reviewing  the  assessments  of  the  Joint 
Staff  and  Breakfast  Club,  DUSD(AT)  makes  the  selection.  The  ACTDs 
initiation  and  approval  process  might  typically  occur  as  described  in 
Appendix  C. 

The  "Guidelines  for  the  ACTD  Management  Plan,"  January  1995,  published  by 
DUSD(AT),  states  that  the  management  plan  is  the  principal  management  tool 
for  the  ACTDs.  As  such,  the  management  plan  is  signed  by  all  participants, 
including  the  developer  and  user,  and  requires  the  identification  of  funding  to 
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complete  ACTDs  and  the  funding  for  supporting  program  efforts  that  are 
essential  to  the  success  of  the  ACTDs.  The  funding  must  be  available  to  ensure 
the  completion  of  the  ACTDs.  In  addition,  the  management  plan  requires  a 
schedule,  which  is  typically  3  years,  to  show  the  critical  events  of  the  ACTDs. 

The  ACTDs  "Guidelines  for  ACTD  Implementation  Directive,"  November  28, 
1995,  is  useful  to  clarify  roles  and  responsibilities  of  the  various  parties 
executing  the  ACTDs  and  to  provide  unambiguous  top-level  guidance.  The 
directive  is  an  agreement  to  define  the  operational  capability  to  be 
demonstrated,  the  general  approach,  the  agencies  responsible  for  planning  and 
conducting  the  demonstration,  and  the  approximate  funding  and  schedule.  The 
parties  executing  the  ACTD  should  sign  the  directive  at  the  time  that  the 
ACTDs  candidate  is  briefed  to  the  Breakfast  Club  and,  if  approved  as  an 
ACTD,  the  directive  would  represent  formal  initiation  of  the  ACTDs. 

Selection  Criteria.  The  DUSD(AT)  officials  stated  that  they  use  the  following 
selection  criteria  to  evaluate  candidate  projects  for  the  ACTDs  program  and  that 
the  criteria  would  be  included  in  the  Defense  Acquisition  Deskbook.1 

o  Technology  should  be  sufficiently  mature; 

o  The  proposed  program  should  provide  significantly  increased  military 
capability; 

o  The  proposed  program  should  be  likely  to  be  affordable  (ensuring  that 
affordability  is  a  part  of  the  ACTDs); 

o  The  timeframe  from  approval  until  completion  of  the  ACTD 
demonstration  should  typically  be  2  to  4  years; 

o  The  user  should  be  intimately  involved  (but  not  committed  to 
procurement); 

o  The  developer  should  be  ready  to  address  the  program  with  a  plan  that 
covers  all  essential  aspects; 

o  All  parties  should  identify,  understand,  and  accept  risks; 

o  The  respective  Service  or  DoD  agency  should  budget  funding  to 
complete  the  demonstration  program,  subject  to  review  of  progress; 

o  Cost-effective  demonstrations  should  be  focused  on  principal  issues; 
and 


o  Funding  should  be  programmed  to  support  2  years  in  the  field. 


iThe  Defense  Acquisition  Deskbook  is  an  electronic  reference  system  that 
provides  the  DoD  acquisition  community  with  discretionary  guidance  for 
implementing  mandatory  DoD  acquisition  direction. 
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Although  not  specifically  included  as  a  selection  criterion,  DoD  Regulation 
5000. 2-R  states  that  ACTDs  should  meet  urgent  military  needs.  Consequently, 
we  included  that  criterion  in  our  evaluation. 


Approved  Projects 


Of  the  22  approved  projects,  valued  at  about  $4  billion,  we  selected  9,  valued  at 
about  $2.7  billion,  to  include  in  the  review  (Appendix  D).  Table  1  shows  our 
assessment  on  whether  the  nine  projects  met  the  selection  criteria.  We 
evaluated  the  projects  against  only  the  selection  criteria  that  we  deemed  most 
essential  for  the  successful  completion  of  the  ACTDs.  Because  DUSD(AT)  did 
not  clearly  define  the  selection  criteria,  we  applied  our  interpretation  of  the 
selection  criteria  to  the  ACTDs  programs  that  we  reviewed. 


Table  1.  Use  of  ACTDs  Selection  Criteria 

Urgent  User  Adequate 

Mature  Military  Intimately  Timeframe  Funding  Funding 
Project  Name  Technology  Need  Involved  (2  to  4  Years)  Budgeted  {millions') 


High  Altitude  Endurance- 

N 

N 

N 

N 

Y 

$  966* 

Unmanned  Aerial  Vehicle 
(UAV) 

Joint  Countermine 

N 

N 

Y 

Y 

Y 

583* 

Kinetic  Energy  Boost 

Phase  Intercept 

N 

N 

Y 

N 

Y 

40* 

Low  Life-Cycle  Cost 
Helicopter 

Y 

N 

Y 

Y 

Y 

21* 

Rapid  Force  Projection 
Initiative 

N 

N 

Y 

Y 

Y 

647* 

Counter  Proliferation 

Y 

N 

Y 

Y 

Y 

123 

Cruise  Missile  Defense 

Y 

N 

Y 

Y 

Y 

80 

Medium  Altitude 

Endurance  UAV 

Y 

Y 

N 

Y 

Y 

147 

Precision/Rapid  Counter 
Multiple  Rocket  Launch 

Y 

N 

Y 

Y 

Y 

91 

Total 

♦Questionable  projects,  which  total  $2.3  billion. 

Application  of  Criteria.  The  DUSD(AT) 

did  not 

consistently 

$2,698 

apply  the 

selection  criteria  when  approving  the  FYs  1995  and  1996  projects  for  the 
ACTDs  program.  As  a  result,  DUSD(AT)  approved  at  least  five  questionable 
ACTDs  projects,  totaling  $2.3  billion. 

For  the  nine  ACTDs  programs  that  we  examined: 

o  Four  of  the  nine  ACTDs  programs  did  not  use  mature  technology. 
Those  were  the  High  Altitude  Endurance-UAV,  the  Joint  Countermine,  the 
Kinetic  Energy  Boost  Phase  Intercept,  and  the  Rapid  Force  Projection  Initiative. 
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We  did  not  consider  the  technology  to  be  mature  if  the  project  used  models  and 
simulations  to  represent  technology  not  yet  developed  or  software  programs  that 
support  ACTDs  such  as  command,  control,  communications,  computers,  and 
intelligence  (C4I),  for  which  software  development  was  concurrent  with  the 
development  of  programs  that  support  the  ACTDs. 

o  Three  of  the  four  ACTDs  relied  significantly  on  software  or 
C4I  development  and  integration  that  were  not  clearly  based  on  mature 
technology.  The  ACTDs  were  the  Rapid  Force  Projection  Initiative,  the  Joint 
Countermine,  and  the  High  Altitude  Endurance-UAV.  For  example,  the  Rapid 
Force  Projection  Initiative  ACTDs  program  office  is  developing  concurrently 
the  Rapid  Force  Projection  Initiative  software  linkages  and  C4I  components  to 
link  non-line-of-sight  sensors  and  shooters.  The  software  and  C4I  development 
is  significant  and  involves  12  subsystems  that  have  to  be  integrated. 

o  All  four  ACTDs  relied  significantly  on  modeling  and 
simulations  because  supporting  programs  were  not  mature.  For  example,  the 
Joint  Countermine  program  office  developed  the  Joint  Countermine  Operational 
Simulation  subsystem  for  evaluating  emerging  Countermine  systems,  doctrine, 
and  tactics  during  the  ACTDs. 

o  Eight  of  the  nine  ACTDs  did  not  have  a  declared  or  documented 
urgent  military  need.  Those  were  the  High  Altitude  Endurance-UAV,  the  Joint 
Countermine,  the  Kinetic  Energy  Boost  Phase  Intercept,  the  Low  Life-Cycle 
Cost  Helicopter,  the  Rapid  Force  Projection  Initiative,  the  Counter 
Proliferation,  the  Cruise  Missile  Defense,  and  the  Precision/Rapid  Counter 
Multiple  Rocket  Launch.  We  based  our  determination  on  whether  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  documented  the  urgent 
need.  We  were  only  provided  documented  evidence  for  the  Medium  Altitude 
Endurance-UAV,  for  which  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  identified  an  urgent  military  need  and  directed  the  Service  to 
proceed  with  its  development.  We  recognize  that  a  military  need  may  be 
declared  by  certain  DoD  officials,  such  as  the  Chiefs  of  Staff  of  the  Military 
Departments,  the  Chairman  of  the  Joint  Chiefs  of  Staff,  or  the  Commanders-in- 
Chief  of  the  Unified  Commands.  However,  DUSD(AT)  did  not  define  what 
constitutes  an  urgent  military  need  or  who  may  declare  the  urgent  need  for  the 
ACTDs  candidates. 

o  Two  of  the  nine  ACTDs  did  not  have  a  user  representative  intimately 
involved  in  the  project.  Those  were  the  High  Altitude  Endurance-UAV  and  the 
Medium  Altitude  Endurance-UAV.  Because  DUSD(AT)  has  not  defined  what 
constitutes  user  involvement,  we  based  our  determination  on  whether  the  user 
had  signed  the  management  plan,  the  implementation  directive,  or  a 
memorandum  of  agreement  (that  is,  whether  the  user  had  a  written  commitment 
to  the  program).  We  recognize  that  the  user  may  be  involved  in  the  ACTDs 
although  the  user  may  not  be  officially  signed  onto  the  ACTDs  management 
plan  or  other  agreement  showing  commitment  to  the  program.  For  example, 
the  U.S.  Atlantic  Command  was  the  designated  user  representative  for  the 
Medium  Altitude  Endurance-UAV  but  had  not  officially  signed  on  to  the  project 
when  DUSD(AT)  approved  it. 
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o  Two  of  the  nine  ACTDs  could  not  be  completed  within  the  preferred  2-  to  4- 
year  timeframe:  the  High  Altitude  Endurance-UAV  and  the  Kinetic  Energy 
Boost  Phase  Intercept.  We  based  our  determination  on  whether  the  ACTDs' 
time  to  complete  as  shown  in  the  management  plan  was  within  that  timeframe. 
For  example,  the  schedule  for  the  Kinetic  Energy  Boost  Phase  Intercept  project 
was  almost  6  years,  which  did  not  include  the  users'  2-year  evaluation.  A  goal 
of  the  ACTD  program  is  to  get  a  critical  technology  into  the  hands  of  the  user 
much  quicker  than  a  streamline  acquisition  process  could  accomplish. 

Questionable  Projects.  We  questioned  five  of  the  nine  projects  as  acceptable 
ACTDs  projects  and  two  of  the  projects  as  marginally  acceptable  based  on  our 
definition  of  the  criterion.  In  addition,  the  Joint  Staff  questioned  several  of  the 
projects. 

Audit  Questioned  Projects.  Of  the  nine  projects  that  we  examined, 
five  ACTDs  projects,  valued  at  $2.3  billion,  were  questionable  for  the  ACTDs 
program.  We  considered  four  projects  questionable  because  two  critical  criteria 
were  not  met:  mature  technology  and  urgent  military  need.  The  four  projects 
that  did  not  meet  those  criteria  are:  the  High  Altitude  Endurance-UAV,  the  Joint 
Countermine,  the  Kinetic  Energy  Boost  Phase  Intercept,  and  the  Rapid  Force 
Projection  Initiative.  We  also  questioned  the  Low  Life-Cycle  Cost  Helicopter 
project  because  it  was  not  advancing  technology  to  meet  a  military  need;  rather, 
the  project  was  to  demonstrate  the  viability  of  commercial  helicopters  to  meet 
the  Navy  requirements,  which  is  an  acquisition  strategy.  In  addition,  the 
Warfighting  Capability  Assessment  team  rated  the  Helicopter  overall  as  low 
potential  to  meet  the  Joint  Warfighting  Capability  Assessment  team  evaluation 
criteria  and  need.  We  identified  two  projects  that  we  considered  marginal 
because  they  did  not  meet  the  urgent  military  need  criterion:  the  Counter 
Proliferation  and  the  Precision/Rapid  Counter  Multiple  Rocket  Launch  ACTDs. 
Appendix  E  describes  how  the  questionable  and  marginal  projects  did  not  meet 
the  criteria. 

We  recognize  that  flexibility  is  essential  in  assessing  how  a  project  meets  a 
specific  criterion  and  that  any  one  project  may  not  meet  all  criteria.  However, 
we  believe  that  if  the  project  did  not  meet  any  of  the  critical  criteria,  its 
selection  as  an  ACTD  was  highly  questionable.  Others  that  met  only  one  or  two 
of  those  criteria  may  be  marginal  candidates,  and  the  approving  body  should 
clearly  state  the  reasons  that  it  approved  the  projects  even  though  they  did  not 
meet  the  critical  criteria. 

Joint  Staff  Comments.  Joint  Staff  officials  presented  their  ratings  of 
the  FYs  1995,  1996,  and  1997  ACTDs  candidates  in  different  ways  to  the 
Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  because 
the  evaluation  process  was  evolving.  During  FY  1995,  Joint  Staff  officials 
discussed  their  results  with  DUSD(AT),  and  in  FY  1996,  the  Joint  Staff  sent  a 
memo  to  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
identifying  that  all  Joint  Requirements  Oversight  Council  principals  were 
supportive  of  the  proposed  candidates.  In  FY  1997,  the  Joint  Staff  officials  sent 
a  memo  to  the  Under  Secretary  ranking  the  candidates  in  order  of  priority  to  the 
Joint  Staff's  needs. 
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The  Joint  Staff  officials  evaluated  the  10  FY  1995  ACTD  projects,  concluding 
that  3  of  the  10  candidates  had  low  potential  as  ACTD  projects.  They  based 
their  assessment  on  their  evaluation  criteria  (Appendix  C),  which  differ  from  the 
ACTDs  selection  criteria  primarily  in  that  the  Joint  Staff  focus  is  on  the 
responsiveness  to  the  need  while  the  ACTDs  definition  focus  is  on  urgent 
military  need.  The  projects  are  the  Synthetic  Theater  of  War,  the  Advanced 
Joint  Planning,  and  the  Rapid  Force  Projection  Initiative.  Additionally,  a  Joint 
Staff  official  stated  that  the  Rapid  Force  Projection  Initiative  was  "colored  by 
uncertainty  with  the  technology,”  specifically  the  Enhanced  Fiber  Optic  Guided 
Missile  subsystem.  Although  Joint  Staff  officials  concluded  that  those  projects 
had  low  potential  to  meet  their  evaluation  criteria  and  need,  DUSD(AT) 
approved  all  three  as  FY  1995  ACTDs. 

Joint  Staff  officials  evaluated  the  12  FY  1996  ACTD  projects,  concluding  that 
the  Miniature  Air-Launched  Decoy  candidate  lacked  technical  maturity.  They 
also  concluded  that  the  Counter  Proliferation  and  the  Air  Base  and  Port 
Biological  Detection  candidates  needed  further  development  to  clarify  utility  to 
the  warfighter.  Additionally,  they  concluded  that  the  Low  Life-Cycle  Cost 
Helicopter  and  the  Land  (Combat)  Vehicle  Survivability  candidates  were  the 
least  well  defined.  Although  Joint  Staff  officials  concluded  that  the  projects' 
technologies  were  not  mature,  lacked  clear  military  utilities,  or  were  not  well 
defined,  DUSD(AT)  approved  all  six  as  FY  1996  ACTDs. 

Dissemination  of  Selection  Criteria.  The  DUSD(AT)  presented  the  selection 
criteria  to  DoD  Components  and  Office  of  the  Secretary  of  Defense  staff 
through  briefings  and  in  an  undated  pamphlet,  "Advanced  Concept  Technology 
Demonstration,  Today's  Technology  for  the  Warfighter."  However,  many 
within  the  DoD  Components  felt  that  the  criteria  and  guidance  for  selecting 
programs  were  unclear.  As  a  result,  of  the  22  projects  selected  for  FYs  1995 
and  1996,  only  four  were  proposed  by  the  Army  and  three  by  the  Navy.  The 
Air  Force  did  not  participate  in  the  FY  1995  and  FY  1996  candidate  proposal 
process  because  of  a  lack  of  guidance. 

As  a  result  of  our  discussion  with  DUSD(AT)  officials,  they  agreed  to  add  the 
selection  criteria  to  the  Defense  Acquisition  Deskbook  (the  Deskbook).  The 
September  1996  update  to  the  Deskbook  contains  the  criteria.  We  believe  that 
adding  the  criteria  will  make  more  people  aware  of  the  ACTD  program  and  will 
provide  proponents  of  potential  ACTD  projects  the  guidance  needed  to  propose 
candidate  programs. 


Defined  Criteria  and  Requirements 


Two  of  the  significant  criteria  for  selecting  an  ACTD  project  are  the  maturity  of 
the  technology  and  the  urgency  of  the  military  need.  Those  criteria  are  found  in 
the  purpose  of  ACTD  as  expressed  in  DoD  Regulation  5000.2-R,  to 
demonstrate  "the  use  of  mature  technology  to  address  urgent  military  needs 
[emphasis  added]."  Although  DUSD(AT)  has  addressed  key  issues  to  improve 
the  process,  such  as  putting  selection  criteria  and  guidance  on  the  ACTD 
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process  in  the  Deskbook,  he  has  not  adequately  defined  mature  technology  and 
urgent  military  need.  Various  DoD  officials,  to  include  a  member  of  the 
Breakfast  Club,  stated  that  they  need  definitive  criteria,  with  some  minimum 
standards,  to  define  what  ACTDs  are  and  to  ensure  that  DUSD(AT)  approves 
only  projects  that  are  appropriate  for  the  ACTD  program. 

Mature  Technology.  DoD  officials  stated  that  technology  supporting  ACTDs 
should  be  mature  or  proven,  and  if  not,  the  technology  should  not  be  included 
in  the  ACTD  program.  Of  the  nine  projects  we  reviewed,  four  did  not  have 
mature  technology  when  we  applied  our  interpretation  of  mature  technology,  as 
described  in  Appendix  E. 

Differing  Definitions.  DoD  officials  defined  mature  technology  in 
widely  contrasting  ways.  For  example,  some  officials  defined  mature 
technology  as: 


o  technology  for  which  a  prototype  already  exists, 

o  technology  that  had  a  risk  level  that  is  acceptable  to  the 
program  manager,  or 

o  technology  that  is  proven. 

Officials  from  the  Office  of  the  Director,  Operational  Test  and  Evaluation, 
stated  that  programs  should  not  be  included  in  the  ACTD  program  if  they  use 
models  and  simulations  to  represent  technology  not  yet  developed.  Also, 
officials  stated  that  DoD  should  not  use  immature  software  programs  that 
support  ACTDs,  such  as  C4I.  Because  of  the  contrasting  definitions,  we 
requested  DUSD(AT)  to  provide  us  the  definition  of  mature  technology.  He 
responded  that: 

The  primary  reasons  for  stressing  the  use  of  mature  technology  are  to 
focus  the  ACTD  on  the  assessment  of  the  military  utility  rather  than 
on  issues  of  the  technical  maturity,  and  to  allow  programs  to  move 
quickly  into  system  integration  and  demonstration  without  extensive 
development  effort.  The  DUSD(AT)  Breakfast  Club  examines  each 
proposed  ACTD  to  assess  the  maturity  of  the  technologies  involved. 

The  DUSD(AT)  Breakfast  Club  membership  includes  the  senior 
technologists  from  the  three  Services  and  from  OSD  [Office  of  the 
Secretary  of  Defense].  The  review  by  the  DUSD(AT)  Breakfast  Club 
is  a  much  more  effective  way  of  assessing  the  maturity  of  the 
underlying  technology  in  a  proposed  ACTD  than  by  the  application  of 
a  set  of  rules  or  measures.  In  summaiy,  there  is  no  set  definition  for 
mature  technology. 

Inconsistent  Terminology.  In  addition  to  not  defining  mature 
technology,  the  guidance  was  inconsistent  when  it  referred  to  the  maturity  of  the 
technology.  The  DUSD(AT)  guidance,  "Guidelines  for  ACTD  Management 
Plan,"  January  1995,  states  that  ACTDs  included  "maturing"  technology, 
whereas  the  guidance,  "ACTD  Initiation  and  Approval  Process,"  July  1995, 
states  that  ACTDs  included  "mature"  technology. 


9 


The  Advanced  Concept  Technology  Demonstration  Selection  Process 


During  our  audit,  DUSD(AT)  agreed  to  our  recommendation  for  consistent  use 
of  terms  in  the  guidance.  The  DUSD(AT)  revised  the  interim  guidance  to 
reflect  the  consistent  use  of  the  term  "mature  technology."  The  Deskbook 
integrated  product  team  approved  the  revised  guidance  on  August  30,  1996,  for 
inclusion  in  the  Deskbook.  Office  of  the  Secretary  of  Defense  officials  stated 
that  they  plan  to  include  the  change  in  the  Deskbook. 

Need  for  Clearer  Definition.  While  we  recognize  that  DUSD(AT) 
needs  flexibility  in  assessing  the  maturity  of  the  technology,  DoD  needs  good 
guidelines  on  what  constitutes  mature  technology  for  an  ACTD  candidate. 
Without  a  clear  definition  of  mature  technology,  DoD  Components  are  left  to 
openly  interpret  the  meaning  of  mature  technology.  Additionally,  without  a 
clear  and  assessable  definition  of  mature  technology,  the  Joint  Staff  officials, 
the  Breakfast  Club,  and  DUSD(AT)  cannot  effectively  and  consistently  assess 
whether  a  proposed  project  for  the  ACTD  program  is  technically  mature  and 
appropriate.  The  Deskbook  should  address  the  definition  of  mature  technology 
in  its  guidance  on  ACTDs.  Such  a  definition  might  be  technology  that  is 
already  fielded  and  needs  no  additional  development  except  when  integrating 
other  fielded  systems.  We  see  those  criteria  in  ACTDs  like  the  Medium 
Altitude  Endurance-UAV  or  the  Cruise  Missile  Defense,  for  which  the  ACTD 
used  technology  that  was  already  fully  developed  (Appendix  D).  While 
specifically  defining  mature  technology  may  be  difficult,  DoD  needs  some 
definition  or  description.  Often  examples  of  the  programs  that  meet  the  criteria 
and  the  programs  that  do  not  meet  the  criteria  will  clarify  the  criteria  definition. 

Urgent  Military  Need.  DoD  officials  have  not  defined  urgent  military  need  or 
declared  that  eight  of  the  nine  approved  ACTDs  met  an  urgent  military  need  as 
described  in  Appendix  E. 

Definition.  The  DoD  Regulation  5000. 2-R  states  that  ACTDs  are  to 
meet  an  "urgent"  military  need;  however,  it  does  not  define  an  urgent  military 
need.  The  published  guidance  of  DUSD(AT)  also  does  not  define  urgent 
military  need  and  does  not  identify  who  may  declare  that  ACTDs  meet  an 
urgent  military  need.  Further,  the  Joint  Staff  evaluation  criteria  do  not  define 
or  describe  the  urgency  of  the  need  for  ACTD  programs.  The  Joint  Staff's 
criteria  for  evaluating  proposed  projects  are  in  Appendix  C.  The  Joint  Staffs 
criteria  are  different  from  DUSD(AT)  selection  criteria  in  that  the  criteria  focus 
on  the  military  need  and  usefulness.  The  Joint  Staff  officials  prioritized  the 
projects  with  respect  to  actual  need.  However,  the  Breakfast  Club  prioritized 
the  projects  with  respect  to  available  funding.  Therefore,  the  issue  of  whether 
ACTDs  approved  by  DUSD(AT)  address  critical  military  needs  is  unclear. 

Only  one  of  the  nine  approved  ACTDs  that  we  evaluated  was  based  on 
an  urgent  military  need.  In  his  July  12,  1993,  memorandum,  the  Under 
Secretary  of  Defense  (Acquisition)2  identified  an  "urgent  need"  for  the 
capability  of  an  endurance  UAV  and  cited  the  characteristics  of  the  Medium 
Altitude  Endurance-UAV  ACTD.  The  DUSD(AT)  and  other  DoD  officials 
provided  no  justifications  of  "urgent  need"  for  any  other  approved  ACTD.  The 


2Renamed  Under  Secretary  of  Defense  for  Acquisition  and  Technology  in 
November  1993. 
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Joint  Staff  officials  identified  four  of  the  nine  as  having  high  potential  in 
meeting  the  Joint  Staff's  need.  The  Joint  Staff  officials  stated  that  they  were 
generally  supportive  of  the  FY  1996  candidates.  However,  input  to  the  Joint 
Staff  ranged  from  supportive,  to  needs  more  work,  to  disagree  for  the  two  FY 
1996  ACTDs  that  we  selected  for  review.  Table  2  shows  the  Joint  Staff's 
prioritization  of  the  nine  projects  that  we  selected  for  review. 

Table  2.  Joint  Staff  Prioritization 


Proiect  Name 

High 

Medium 

Low 

High  Altitude  Endurance-UAV 

X 

Joint  Countermine 

X 

Kinetic  Energy  Boost  Phase  Intercept 
Low  Life-Cycle  Cost  Helicopter 

X 

X 

Rapid  Force  Projection  Initiative 
Counter  Proliferation 

Cruise  Missile  Defense 

X 

X 

X 

Medium  Altitude  Endurance-UAV 
Precision/Rapid  Counter 

X 

Multiple  Rocket  Launch 

X 

Consistency.  DUSD(AT)  guidance  was  inconsistent  when  it  referred  to 
ACTDs  meeting  military  needs.  For  example,  "ACTD  Initiation  and  Approval 
Process"  states  that  ACTDs  solve  "important  military  problems,"  whereas 
"Guidelines  for  ACTD  Implementation  Directive,"  November  1995,  states  that 
ACTDs  meet  "urgent  military  operational  needs."  Although  inconsistent  in 
terminology,  both  documents  were  consistent  in  stating  that  ACTDs  should 
improve  a  military  capability.  During  our  audit,  DUSD(AT)  agreed  to  our 
recommendation  for  consistent  use  of  terms  in  its  guidance.  DUSD(AT) 
officials  revised  the  interim  guidance  and  changed  the  terminology  from 
"urgent"  to  "critical."  The  Deskbook  integrated  product  team  approved  the 
change  on  August  30,  1996.  Office  of  the  Secretary  of  Defense  officials  plan  to 
incorporate  the  change  in  the  Deskbook.  While  those  actions  correct  the 
inconsistent  use  of  the  term,  the  definition  of  critical  military  need  is  still 
unclear. 

We  agree  that  ACTDs  should  meet  critical  military  needs  and  improve  military 
capabilities.  However,  without  a  clear,  Defense-wide  definition  of  a  critical 
military  need,  the  definitions  will  vary  greatly  among  DoD  Components. 
Additionally,  the  Joint  Staff,  the  Breakfast  Club,  and  DUSD(AT)  cannot 
effectively  and  consistently  assess  whether  an  ACTD  candidate  meets  a  critical 
military  need  and  is  appropriate  for  the  ACTD  program  without  a  definition. 
All  criteria  must  be  clear  and  assessable,  whether  stated  qualitatively  or 
quantitatively.  Reasonable  criteria  are  those  that  informed  persons  could  use  to 
reach  substantially  similar  conclusions.  An  example  of  a  potential  clear 
definition  of  critical  military  need  is  a  need  that  the  Joint  Staff  identifies  as  high 
potential  to  meet  the  Joint  Warfighting  Capability  Assessment  team  evaluation 
criteria  and  need.  Further,  a  qualitative  or  quantitative  criterion  should  have  a 
clear  definition,  free  of  unclear  or  imprecise  terminology  such  as  "adequate, 
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sufficient,  or  acceptable."  The  DUSD(AT)  should  clearly  define  those  criteria, 
provide  the  definitions  to  include  examples  in  the  Deskbook,  and  identify  which 
DoD  officials  may  declare  that  an  ACTD  meets  a  critical  military  need. 


Support  for  ACTD 


As  stated  earlier,  because  of  confusion  about  the  program  and  the  lack  of  clarity 
in  the  Office  of  the  Secretary  of  Defense  guidelines  and  selection  criteria,  the 
Army  and  the  Navy  were  the  only  Services  that  proposed  ACTD  candidates. 
Some  DoD  officials  did  not  view  technology  used  in  some  ACTDs  as  mature. 
Others  stated  that  ACTD  programs  should  not  include  software  and  C4I  systems 
development  and  integration,  and  modeling  and  simulations  used  to  represent 
technology,  because  those  elements  do  not  represent  mature  technology.  The 
ambiguity  contributed  to  the  lack  of  broad  support  within  DoD. 

In  addition,  Congress  is  concerned  about  the  program.  In  the  National  Defense 
Authorization  Act  for  Fiscal  Year  1997,  July  30,  1996,  the  conferees  stated  that 
"it  appears  that  the  Department  is  using  the  ACTD  program  to  circumvent 
acquisition  requirements,  rather  than  to  demonstrate  new  technologies  on  a 
limited  basis."  The  conferees  specifically  cited  the  Joint  Tactical  UAV  project, 
an  FY  1997  ACTD  candidate.  Additionally,  the  Senate  Committee  on 
Appropriations  stated  that  the  complex  development  required  for  some  ACTDs 
may  not  be  appropriate  for  the  streamlined  acquisition  procedures  used  for 
ACTDs.  The  Joint  Countermine  is  an  example  of  a  very  complex  development. 
The  complexity  is  driven  by  the  need  to  integrate  14  Army,  Navy,  and  Marine 
Corps  technical  demonstration  projects  and  field  military  equipment.  The 
projects  include  advanced  technology  demonstration  and  concept 
demonstrations,  which  are  programs  still  in  development.  Additionally,  a 
significant  portion  of  the  project  is  the  development  of  software  and  C4I  to 
integrate  those  subprojects. 


Management  Actions 


Implementation  of  the  ACTD  process  has  been  evolving  since  its  inception  in 
1994.  The  Military  Departments  nominated  7  of  the  FYs  1995  and  1996 
approved  ACTDs  and  nominated  5  of  the  25  final  candidates  for  FY  1997.  In 
addition,  Office  of  the  Secretary  of  Defense  officials  have  made  changes  to 
much  of  their  guidance  and  required  the  Breakfast  Club  to  document  its 
deliberation  of  candidate  ACTD  projects. 

Criteria  Defined.  Office  of  the  Secretary  of  Defense  officials  recognized  that 
the  ACTD  program  is  new  to  the  Department  and  that  program  changes  will 
occur  because  of  lessons  learned.  During  the  audit,  officials  recognized  that 
some  criteria  for  the  selection  of  projects  for  the  ACTD  program  were  not 
clearly  or  consistently  stated  among  the  guidance  documents.  Officials  took 
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corrective  action  and  plan  to  include  the  changes  in  the  Deskbook.  However,  as 
discussed  earlier,  mature  technology  and  critical  military  need  still  need 
defining.  Also,  Office  of  the  Secretary  of  Defense  officials  recognized  that  the 
policies  and  procedures  for  transitioning  into  the  Defense  acquisition  cycle  were 
not  adequate,  and  they  published  interim  guidance.  We  commend  Office  of  the 
Secretary  of  Defense  officials  for  recognizing  the  need  for  change  and  for  taking 
steps  necessary  to  improve  the  ACTD  program. 

Documented  Evaluations.  The  Joint  Staff  and  the  Breakfast  Club  did  not 
document  deliberations  in  evaluating  the  candidates  and  in  showing  how  they 
reached  the  conclusions  and  recommendations  to  DUSD(AT)  based  on  the 
selection  criteria.  Also,  no  documentation  existed  that  showed  how  DUSD(AT) 
evaluated  the  candidates  with  respect  to  the  selection  criteria.  Other  than  the 
initial  briefing  to  the  Breakfast  Club,  the  proponents  provided  no  additional 
information  or  briefings  unless  requested.  In  addition,  none  of  the  participants 
retained  meeting  minutes  for  the  FYs  1995  and  1996  candidate  projects. 

During  our  meeting  in  December  1995,  we  expressed  our  concerns  to 
DUSD(AT)  about  the  lack  of  documentation  showing  the  consideration  given  to 
the  selection  criteria  for  the  approved  candidates.  In  January  1996,  DUSD(AT) 
directed  the  Breakfast  Club  members  to  document  their  assessments  of  the 
proposed  ACTDs  for  FY  1997,  and  they  did. 


Other  Matters  of  Interest 


Guidance.  The  Office  of  the  Secretary  of  Defense  defined  ACTDs  in  DoD 
Regulation  5000.2-R,  and  stated  that  ACTDs  are  not  acquisition  programs. 
However,  DoD  Regulation  5000.2-R  only  governs  major  Defense  acquisition 
and  major  automated  information  system  acquisition  programs  and  not  other 
Defense  acquisition  programs  such  as  ACTDs.  Because  DoD  Directive  5000. 1, 
"Defense  Acquisition,"  applies  to  all  DoD  acquisition  programs,  Office  of  the 
Secretary  of  Defense  officials  should  include  information  on  ACTDs  in  that 
directive  rather  than  in  the  regulation.  During  our  meeting  in  August  1996, 
DUSD(AT)  agreed. 

ACTD  Management  Plans.  The  DUSD(AT)  did  not  meet  its  goal  to  have 
management  plans  for  the  projects  approved  in  a  timely  manner.  Although  the 
management  plans  are  not  part  of  the  selection  criteria,  ACTD  program 
managers  require  them  to  effectively  manage  their  respective  project.  The 
DUSD(AT)  guidance,  "ACTD  Initiation  and  Approval  Process,"  July  1995, 
states  that  all  parties  should  fully  endorse  the  management  plan  within  60  days 
after  executive  level  approval  of  the  ACTD.  Only  one  of  the  nine  projects 
meets  that  goal.  For  five  of  nine  projects  that  we  examined,  the  approval 
process  averaged  366  days.  Three  of  the  five  projects  were  the  same  projects 
that  we  questioned.  Those  projects  were  the  High  Altitude  Endurance-UAV, 
the  Joint  Countermine,  and  the  Rapid  Force  Projection  Initiative.  Three 
projects  concluded  before  they  had  an  approved  management  plan.  Because 
DoD  and  Component  officials  did  not  approve  the  plans  expeditiously,  program 
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managers  were  forced  to  clarify  responsibilities  and  objectives  as  needed 
through  memorandums  of  agreement,  independent  advisory  groups  and 
committees,  and  the  integrated  product  team  process.  Because  management 
took  actions  to  resolve  timeliness  issues,  this  report  makes  no  recommendations. 

Continual  Funding.  Five  of  the  nine  projects  that  we  examined  did  not  have 
funding  budgeted  to  complete  the  planned  demonstration,  to  include  the  users' 
2-year  evaluation.  Those  projects  were  the  High  Altitude  Endurance-UAV,  the 
Joint  Countermine,  the  Kinetic  Energy  Boost  Phase  Intercept,  the  Rapid  Force 
Projection  Initiative,  and  the  Precision/Rapid  Counter  Multiple  Rocket  Launch. 
The  funding  problems  occurred  after  DUSD(AT)  selected  the  project  for  the 
ACTD  program.  The  ACTD  and  its  supporting  programs  had  to  compete  with 
other  needs  of  the  Services.  We  based  our  determination  on  whether  the  ACTD 
and  its  supporting  programs  were  funded  at  the  time  we  conducted  our  audit. 
We  believe  that  if  the  ACTD  projects  truly  had  a  critical  military  need  and  the 
Services  fully  supported  the  ACTD  program  by  nominating  candidates,  then  the 
Services'  priority  for  the  projects  would  have  been  higher  and  funding  more 
likely.  For  example,  the  program  office  did  not  fully  fund  key  elements  of  the 
Precision/Rapid  Counter  Multiple  Rocket  Launch  project,  such  as  the 
Reconnaissance  Infrared  Surveillance  Target  Acquisition  System,  totaling  about 
$9  million. 

We  recognize  that  all  programs  must  compete  for  dollar  resources.  However, 
programs  that  are  to  meet  a  critical  military  need  within  a  2-  to  4-year 
timeframe  must  maintain  their  initial  funding  profile  if  they  are  to  succeed. 
When  DUSD(AT)  clarifies  the  selection  criteria,  he  should  consider  only  those 
projects  that  meet  a  critical  military  need  and  other  defined  criteria  for  the 
ACTD  program  and  accordingly  will  fund  those  projects  because  of  their  critical 
need  to  the  warfighter. 


Conclusion 


The  processes  for  selecting  and  approving  projects  for  the  ACTD  program  are 
maturing,  and  DUSD(AT)  is  resolving  issues  and  problems  as  they  arise. 
However,  the  processes  for  selecting  and  approving  projects  could  be  more 
effective  with  guidance  that  clearly  defines  the  program's  selection  and  approval 
criteria  for  mature  technology  and  critical  military  need.  Clarifying  the  criteria 
will  result  in: 

o  the  selection  process  being  more  explicit  and  understood; 

o  consistent  application  of  those  criteria  to  ensure  that  DUSD(AT) 
approves  only  projects  that  are  appropriate  for  the  ACTD  program; 
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o  solidifying  support  within  Congress,  DoD,  and  the  Services  for  the 
ACTD  program  as  the  ACTD  candidates,  approved  programs,  and  dollars 
allocated  and  spent  continue  to  grow  in  the  unique  application  of  technology; 
and 


o  decreased  potential  of  program  failure. 


Management  Comments  and  Audit  Response  to  the  Finding 


The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  provided 
extensive  comments  on  the  report's  content.  The  Under  Secretary  disagreed 
with  our  interpretation  of  the  criteria,  specifically  those  key  criteria  of  military 
need  and  mature  technology.  However,  the  Under  Secretary  stated  that  the 
criteria  should  be  clarified  to  allow  better  understanding  by  those  not  directly 
involved  in  the  selection  process. 

We  developed  our  definition  because  the  criteria  were  not  clearly  understood, 
and  there  were  different  interpretations  of  the  criteria.  We  believe  that  the 
criteria  should  be  also  clarified  for  those  involved  in  the  selection  process  as 
well  as  for  other  interested  parties.  As  personnel  involved  in  the  selection 
process  change,  the  clarification  of  the  guidance  will  ensure  consistency  in  the 
process  and  in  the  application  of  the  criteria.  We  made  appropriate  changes  to 
the  finding  based  on  the  comments  and  responded  to  the  Under  Secretary's 
comments  in  Appendix  F.  The  full  text  of  his  comments  are  in  Part  III  of  this 
report. 


Recommendations,  Management  Comments  and  Audit 
Response 

We  recommend  that  the  Deputy  Under  Secretary  of  Defense  (Advanced 
Technology): 

1.  Establish  a  working  group  consisting  of  members  from  the  user, 
developer,  tester,  and  Office  of  the  Secretary  of  Defense  communities  to 
develop  clear  and  assessable  selection  criteria,  including  the  definitions  of 
mature  technology  and  critical  military  need,  and  include  the  revised 
selection  criteria  in  the  Defense  Acquisition  Deskbook. 

2.  Reevaluate  the  FY  1997  candidates  once  the  selection  criteria  are 
developed,  to  ensure  that  they  meet  the  assessable  selection  criteria  defined 
by  the  working  group. 
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The  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Comments.  The  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
concurred,  stating  that  the  intent  of  the  selection  criteria  needs  clarification  and 
that  the  Department  will  reexamine  the  ACTDs  selected  for  FY  1997  to  ensure 
that  the  candidates  are  consistent  with  the  intent  of  the  selection  criteria.  We 
will  be  pleased  to  participate  in  a  working  group,  as  he  suggested,  to  clarify  the 
criteria. 

The  Vice  Chairman  of  the  Joint  Chief  of  Staff  Comments.  Although  not 
required  to  comment,  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  provided 
comments  on  the  finding  and  recommendations.  The  Vice  Chairman  stated  that 
while  he  found  fault  with  some  initial  assumptions  and  some  specific 
observations,  he  endorses  the  recommendations.  For  the  full  text  of  the  Vice 
Chairman's  comments,  see  Part  III. 
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n 


Appendix  A.  Scope  and  Methodology 


Scope 


We  limited  our  scope  to  the  22  approved  ACTD  projects  for  FYs  1995  and 
1996  as  shown  in  Appendix  B.  We  judgmentally  selected  nine  projects,  valued 
at  $2.7  billion,  for  evaluation.  The  nine  projects  are  listed  and  described  in 
Appendix  D.  The  sample  projects  were  selected  to  get  projects  representing: 

o  various  stages  of  development, 

o  single  systems, 

o  systems-of-systems, 

o  concepts  of  operations, 

o  management  of  single  components,  and 

o  management  of  multiple  components. 


Methodology 


We  conducted  this  economy  and  efficiency  audit  from  October  1995  through 
October  1996  in  accordance  with  auditing  standards  issued  by  the  Comptroller 
General  of  the  United  States,  as  implemented  by  the  Inspector  General,  DoD, 
and  included  such  tests  of  management  controls  as  deemed  necessary. 

We  reviewed  data  from  December  1993  through  August  1996.  We  evaluated 
the  policies  and  procedures  of  DUSD(AT)  and  the  DoD  Components  for 
selecting,  developing,  and  executing  ACTD  projects.  We  compared  and 
analyzed  the  process  for  selection  of  the  ACTDs,  the  process  for  determining 
effectiveness  of  the  effort,  and  the  transition  of  ACTDs  into  the  Defense 
acquisition  cycle.  The  audit  did  not  rely  on  computer-processed  data.  We 
interviewed  Office  of  the  Secretary  of  Defense  and  DoD  Component  officials. 

No  prior  audits  covered  the  ACTD  program. 
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Management  Control  Program 

DoD  Directive  5010.38,  "Internal  Management  Control  Program,"  April  14, 
1987,*  requires  DoD  organizations  to  implement  a  comprehensive  system  of 
management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  Review  of  Management  Control  Program.  We  evaluated  the 
adequacy  of  the  DUSD(AT)  and  the  DoD  Components'  management  controls 
over  the  selection,  management,  and  transition  into  the  acquisition  cycle  of 
ACTD  programs.  In  assessing  those  controls,  we  evaluated  plans  and 
procedures,  written  policies,  and  management-initiated  reviews. 

Adequacy  of  Management  Controls.  The  policies  and  procedures  governing 
the  effectiveness  of  the  process  and  the  transition  of  programs  into  the 
acquisition  cycle  were  generally  adequate.  However,  we  identified  a  material 
management  control  weakness  as  defined  by  DoD  Directive  5010.38.  The 
DUSD(AT)  needs  to  establish  management  controls  to  define  and  consistently 
apply  criteria  for  evaluating  and  selecting  projects  for  the  ACTD  program.  See 
Part  I  for  details  on  the  material  weakness.  The  recommendations,  if 
implemented,  will  improve  the  effectiveness  and  efficiency  of  the  ACTD 
program.  Specifically,  the  lack  of  clearly  defined  criteria  led  to  the  inconsistent 
application  of  the  criteria  to  select  and  approve  ACTD  candidates.  A  copy  of 
the  report  will  be  provided  to  the  senior  official  responsible  for  management 
controls  in  the  Office  of  the  DUSD(AT)  and  the  Under  Secretary  of  Defense 
(Comptroller). 

Adequacy  of  Management's  Self-Evaluation.  The  Office  of  the  Deputy 
Under  Secretary  of  Defense  (Advanced  Technology)  and  the  DoD  Components 
did  not  establish  ACTD  projects  as  assessable  units.  Accordingly,  they  did  not 
identify  or  report  the  material  management  control  weakness  identified  by  the 
audit. 


Area  Not  Requiring  Further  Review 


Effectiveness  of  the  ACTD  Projects.  We  met  with  users,  ACTD  project 
managers,  and  senior  DoD  officials  to  evaluate  whether  the  process  for 
determining  the  effectiveness  of  the  projects  was  adequate.  We  also  reviewed 
the  projects'  management  plans,  the  results  of  in-process  reviews,  and  other 
program  documentation.  The  process  that  the  Office  of  the  Secretary  of 


*DoD  Directive  5010.38  has  been  revised  as  "Management  Control  (MC) 
Program,"  August  26,  1996.  The  audit  was  performed  under  the  April  1987 
version  of  the  directive. 
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Defense  officials  used  for  determining  the  effectiveness  of  the  ACTDs  was 
working.  Both  the  oversight  committees  and  the  integrated  product  teams 
addressed  and  resolved  issues  when  identified. 

Transition  to  Acquisition.  Office  of  the  Secretary  of  Defense  officials 
recognized  that  the  guidance  for  assessing  whether  the  project  met  military 
needs  and  planning  for  transition  into  the  acquisition  cycle  were  not  sufficient. 

Assessing  Military  Utility.  The  DUSD(AT)  issued  interim  guidance 
in  January  1995  that  required  meaningful,  quantitative  measures  of  success  to 
evaluate  whether  the  project  would  meet  military  needs.  Those  measures  of 
success  included  measures  of  effectiveness  and  measures  of  performance. 
Although  the  early  guidance  did  not  detail  measuring  success,  the  DoD 
Components  and  Office  of  the  Secretary  of  Defense  officials  assessed  the 
project's  success  through  an  iterative  process,  which  saved  time  and  money. 
For  example,  after  the  Precision/Rapid  Counter  Multiple  Rocket  Launch 
approval  by  the  DUSD(AT)  as  an  ACTD,  the  project  office  saved  $16  million 
by  canceling  the  Modified  Dual-Purpose  Improved  Conventional  Munition,  a 
program  that  supported  the  ACTD,  because  the  program  did  not  meet  one  of  the 
Precision/Rapid  Counter  Multiple  Rocket  Launch  requirements. 

Acquisition  Planning.  On  July  21,  1995,  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  directed  that  transition  planning  should 
start  for  the  ACTDs  initiated  in  FY  1995  and  that  integrated  product  teams 
should  accomplish  the  planning  effort.  Additionally,  the  DUSD(AT)  developed 
templates  for  project  managers  to  use  to  smoothly  transition  their  efforts  into  the 
acquisition  process.  The  templates  address  both  assessing  military  utility  and 
transitioning  to  the  acquisition  cycle.  Office  of  the  Secretary  of  Defense 
officials  plan  to  include  the  draft  of  their  interim  guidance,  "Transition  of 
Advanced  Concept  Technology  Demonstration  (ACTD)  to  the  Formal 
Acquisition  Process,"  June  19,  1996,  in  the  Deskbook. 


Organizations  and  Individuals  Visited  or  Contacted 


We  visited  or  contacted  individuals  and  organizations  within  DoD;  the  General 
Accounting  Office;  and  the  Applied  Physics  Laboratory,  Johns  Hopkins 
University.  Further  details  are  available  on  request. 
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Approved  ACTDs  for  FY  1995 


Amount 

(millions) 

High  Altitude  Endurance-UAV  $  966 

Rapid  Force  Projection  Initiative  647 

Joint  Countermine  583 

Synthetic  Theater  of  War  439 

Advanced  Joint  Planning  222 

Medium  Altitude  Endurance  Unmanned  Aerial  Vehicle  147 

Precision/Rapid  Counter  Multiple  Rapid  Launch  91 

Cruise  Missile  Defense  80 

Precision  Signal  Intelligence  Targeting  44 

Kinetic  Energy  Boost  Phase  Intercept  40 


Total  for  FY  1995  Projects  $3,259 


Approved  ACTDs  for  FY  1996 

Amount 

(millions) 

Tactical  Unmanned  Aerial  Vehicle  $145 

Combat  Identification  126 

Counter  Proliferation  123 

Battlefield  Awareness  Data  Dissemination  85 

Semi  Automated  Image  Processing  63 

Navigation  Warfare  57 

Miniature  Air-Launched  Decoy  32 

Combat  Vehicle  Survivability  22 

Low  Life-Cycle  Cost  Helicopter  21 

Air  Base  and  Port  Biological  Detection  19 

Joint  Logistics  13 

Joint  Readiness  Extension  to  the  Advanced  Joint  Planning  _ 6 

Total  for  FY  1996  Projects  $712 
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Rework 


JROC  Joint  Requirements  Oversight  Council 

JWCA  Joint  Warfighting  Capability  Assessment 


User  and  Developer  Team.  A  crucial  ground  rule  for  any  project  proposed  for 
the  ACTD  program  is  the  need  for  a  close  partnership  between  a  sponsoring 
user  organization  and  a  DoD  Component  acquisition  organization.  The  proposed 
must  identify  and  develop  that  relationship  before  the  DUSD(AT)  gives  any 
serious  consideration  to  the  project.  The  DUSD(AT)  staff  is  to  assist  the  team 
of  user  and  developer  by  clarifying  the  selection  criteria  and  team 
responsibilities  and  is  to  assist  in  developing  and  refining  the  proposal. 

Initial  Briefing  to  DUSD(AT).  Once  the  team  defines  its  proposal,  the  team 
presents  it  to  the  DUSD(AT).  The  DUSD(AT)  reviews  the  proposal  for 
military  need,  potential  capability,  and  affordability.  The  DUSD(AT)  then 
either  accepts  the  proposal  for  further  discussion  as  a  formal  ACTD  candidate, 
requests  revision  of  the  proposal,  or  determines  that  the  proposal  is  outside  the 
scope  of  the  ACTD  program. 

Review  and  Prioritization.  If  DUSD(AT)  accepts  the  proposal  as  an  ACTD 
candidate,  then  both  the  Joint  Staff  and  the  Breakfast  Club  independently  assess 
the  candidate.  Both  the  Joint  Staff  and  the  Breakfast  Club  provide  their 
assessments  to  the  DUSD(AT)  for  his  final  selection. 
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Joint  Staff  Review  and  Prioritization.  Following  the  initial 
DUSD(AT)  review,  the  DUSD(AT)  forwards  information  on  the  ACTD 
candidates  to  the  Joint  Staff's  Joint  Requirements  Oversight  Council  (the 
Council).  The  Council  then  disseminates  the  list  of  ACTD  candidates  to  the 
Joint  Staff's  Joint  Warfighting  Capability  Assessment  teams.  Each  Joint 
Warfighting  Capability  Assessment  team  evaluates  a  specific  joint  warfighting 
area. 

The  team  assesses  and  prioritizes  the  ACTD  candidates  with  respect  to  the  Joint 
Warfighting  Capability  Assessment  need.  Joint  Staff  officials  developed  six 
evaluation  criteria  for  the  Joint  Warfighting  Capability  Assessment  team  to  use 
to  assess  the  ACTD  candidates.  The  evaluation  criteria  are  as  follows: 

o  Joint  Warfighting  Capability  Assessment  team  linkage  to  the  proposed 

ACTD; 


o  Responsiveness  to  need; 
o  Strength  of  user  sponsorship; 
o  Potential  to  lead  to  permanent  solution; 
o  Value  of  residual  capability;  and 

o  Degree  of  doctrine  or  concept  of  operations,  technology,  and  force 
structure  integration  (or  planning  for  same). 

The  Joint  Warfighting  Capability  Assessment  team  forwards  its  assessments  and 
prioritized  list  to  the  Council.  Besides  the  Joint  Warfighting  Capability 
Assessment  evaluation,  the  Council  obtains  comments  about  the  candidate 
ACTDs  from  the  Unified  Commanders  in  chief.  The  Council  then  prioritizes 
the  candidate  ACTDs  based  on  the  Joint  Warfighting  Capability  Assessment  and 
Unified  Commander's  input.  The  Council  then  forwards  its  prioritized  list  to 
the  DUSD(AT). 

Breakfast  Club  Review.  Concurrent  with  the  Joint  Warfighting 
Capability  Assessment  team  review,  the  developer  and  user  brief  the  Breakfast 
Club  on  the  technical  and  schedule  aspects  of  the  candidates.  Additionally,  the 
Breakfast  Club  reviews  the  candidates  with  respect  to  available  funding.  The 
Breakfast  Club  provides  its  recommendations  to  the  DUSD(AT). 

DUSD(AT)  Approval  of  ACTDs.  Upon  receipt  of  the  assessments  and 
recommendations  of  the  Joint  Staff  and  Breakfast  Club,  the  DUSD(AT)  makes 
the  ACTD  selection  for  the  ACTD  program. 
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Counter  Proliferation.  The  system,  $123  million  funded,  combines  sensors, 
weapon  systems,  and  collateral  damage  prediction  and  bomb  damage  assessment 
tools  to  provide  an  improved  capability  to  detect  and  destroy  hardened 
production  and  storage  facilities  for  weapons  of  mass  destruction.  The  Defense 
Special  Weapons  Agency  and  the  European  Command  manage  the  project 
jointly. 

Cruise  Missile  Defense.  The  system,  $80  million  funded,  combines  an 
airborne  sensor  suite  and  a  surface-based  air  defense  system  to  provide  the 
capability  to  detect  and  destroy  cruise  missiles  at  ranges  beyond  the  radar  line- 
of-sight  of  surface-based  (sea  and  land)  air  defense  units.  The  Army  and  Navy 
manage  the  project  jointly. 

High  Altitude  Endurance-UAV.  The  system,  $966  million  funded,  combines 
sensors  and  data  processors  on  a  UAV  to  operate  in  all  weather,  day  and  night, 
at  high  altitudes  and  to  provide  wide  area,  near  real-time  imagery.  The  project 
consists  of  two  air  vehicles:  the  Global  Hawk,  a  conventional  UAV,  and  the 
Darkstar,  a  low  observable  UAV.  The  Global  Hawk  will  operate  in  low-to- 
moderate  air  defense  threats.  The  Darkstar  will  operate  in  high  air  defense 
threats.  The  Defense  Advanced  Research  Projects  Agency  manages  the  project 
with  assistance  from  the  Army,  the  Navy,  and  the  Air  Force. 

Joint  Countermine.  The  system,  $583  million  funded,  combines  sensors  and 
weapon  systems  designed  to  detect  and  neutralize  sea  and  land  mines  and  to 
provide  the  capability  to  conduct  seamless  amphibious  and  ground  forces  mine 
countermeasure  operations.  The  Army  and  Navy  comanage  the  project. 

Kinetic  Energy  Boost  Phase  Intercept.  The  system,  $40  million  funded, 
combines  sensors,  airborne  interceptor  missiles,  and  battle  management 
command  and  control  systems.  The  system  will  acquire  and  intercept  theater 
ballistic  missiles  in  their  boost  phase  and  early  mid-course  of  flight.  The  draft 
management  plan  states  that  the  ACTD  will  include  two  phases.  Phase  I  will 
use  models  and  simulation  to  verify  that  airborne  interceptor  can  perform 
effectively  with  existing  forces.  Phase  II  will  demonstrate  the  actual  intercept 
of  a  target  with  a  missile.  The  Air  Force  manages  the  project. 

Low  Life-Cycle  Cost  Helicopter.  The  project,  $21  million  funded,  is  to 
evaluate  whether  leasing  commercial  helicopters  will  reduce  shipboard  vertical- 
lift  operating  costs.  The  Navy  manages  the  project. 

Medium  Altitude  Endurance-UAV.  The  system,  $147  million  funded,  known 
as  the  Predator,  combines  sensors  and  data  processors.  The  surveillance  system 
will  operate  in  most  weather  at  medium  altitudes  and  will  provide  sustained, 
near  real  time  intelligence  information  on  fixed  and  mobile  targets.  The 
Unmanned  Aerial  Vehicles  Joint  Project  Office  under  the  Program  Executive 
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Office  for  cruise  missiles  and  unmanned  aerial  vehicles  serves  as  the 
demonstration  manager,  and  the  U.S.  Atlantic  Command  serves  as  the 
operations  manager. 

Precision/Rapid  Counter  Multiple  Rocket  Launch.  The  system,  $91  million 
funded,  combines  sensors,  weapon  control  systems,  and  data  processing  and 
display  to  provide  an  adverse  weather,  day  and  night,  end-to-end,  sensor-to- 
shooter,  precision  deep-strike  capability.  The  Army  manages  the  project. 

Rapid  Force  Projection  Initiative.  The  system,  $647  million  funded, 
combines  sensors  and  weapon  systems  designed  to  detect  and  destroy  targets 
beyond  traditional  direct  fire  ranges.  The  system  provides  increased  lethality 
and  survivability  for  early  entry  forces.  The  Army  manages  the  project. 
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Questionable  Projects 


The  High  Altitude  Endurance-UAV,  Joint  Countermine,  Kinetic  Energy  Boost 
Phase  Intercept,  Low  Life-Cycle  Cost  Helicopter,  and  Rapid  Force  Projection 
Initiative  projects  were  not  appropriate  for  the  ACTD  program  and  are 
questionable. 

High  Altitude  Endurance-UAV.  We  did  not  consider  the  High  Altitude 
Endurance-UAV  appropriate  for  the  ACTD  program  because  the  UAV  was  not 
technically  mature  and  did  not  meet  a  critical  military  need  when  we  applied  our 
interpretation  of  those  selection  criteria  to  the  ACTD.  In  addition,  the  user  was 
not  intimately  involved,  and  the  UAV  was  not  scheduled  for  completion  within 
the  timeframe  of  2  to  4  years. 

Mature  Technology.  The  DUSD(AT)  considered  the  High  Altitude 
Endurance-UAV  project  technically  mature  for  the  ACTD  program.  However, 
one  of  the  two  air  vehicles  that  comprise  the  High  Altitude  Endurance-UAV, 
the  Darkstar,  was  not  technically  mature.  Before  approval  of  the  High  Altitude 
Endurance-UAV  as  an  ACTD,  the  Defense  Advanced  Research  Projects 
Agency,  the  executive  agent  for  the  project,  concluded  that  the  Darkstar' s 
overall  technical  risk  was  "medium  to  high."  The  "Cost  Analysis  Improvement 
Group  Cost  Assessment,"  May  13,  1994,  states  that  the  Darkstar  project  had  a 
substantial  risk  of  cost  growth,  particularly  for  software  development.  The 
price  negotiation  memorandum  for  the  Darkstar,  June  24,  1994,  concluded  that 
development  of  its  software  was  "high  risk. " 

During  the  High  Altitude  Endurance-UAV  approval  process,  the  program  office 
did  not  present  analytical  studies  to  demonstrate  the  technical  maturity  of  the 
Darkstar.  The  contractor  had  not  yet  developed  the  Darkstar*  s  avionics 
software,  which  includes  the  system's  flight  control  software.  Resulting 
software  development  problems  delayed  the  Darkstar's  first  flight  for  6  months. 

The  Darkstar  crashed  upon  takeoff  during  its  second  test  flight  on  April  22, 
1996.  In  its  press  release,  July  8,  1996,  the  Defense  Advanced  Research 
Projects  Agency  stated  that  its  safety  investigation  concluded  that  the  Darkstar 
crashed  as  a  result  of  the  system's  flight  control  software  and  deficiencies  in  the 
system's  modeling  and  simulation.  Subsequently,  the  Defense  Airborne 
Reconnaissance  Office  requested  $29  million  to  replace  the  Darkstar  that 
crashed,  to  configure  a  second  air  vehicle  for  flight,  to  accelerate  fabrication  of 
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a  third  air  vehicle,  to  implement  changes  in  program  process  and  procedures, 
and  to  support  hardware  changes  likely  to  result  from  accident  investigation 
findings. 

Critical  Military  Need.  Although  the  Joint  Staff  rated  the  High 
Altitude  Endurance-UAV  overall  as  high  potential  to  meet  the  Joint  Warfighting 
Capability  Assessment  team's  evaluation  criteria  and  need,  it  did  not  address 
critical  military  need,  and  we  could  not  find  evidence  that  the  Under  Secretary 
of  Defense  for  Acquisition  and  Technology  stated  that  the  project  met  a  critical 
military  need. 

User  Intimately  Involved.  We  based  our  determination  on  whether  the 
user  had  signed  the  implementation  directive  before  the  approval  of  the  ACTD 
or  the  Management  Plan  during  the  ACTD.  The  High  Altitude  Endurance- 
UAV  user  did  not  sign  the  management  plan.  Also,  the  High  Altitude 
Endurance-UAV  did  not  have  an  implementation  directive,  but  used  instead  a 
memorandum  of  agreement,  which  the  user  did  not  sign. 

Timeframe  (2  to  4  Years).  We  determined  whether  the  ACTD  went 
beyond  the  4  years  from  the  time  that  the  DUSD(AT)  approved  the  ACTD  until 
the  completion  of  the  demonstration.  The  DUSD(AT)  approved  the  program  in 
August  1994,  and  the  program  continues  through  the  fourth  quarter  of  1999,  as 
shown  in  the  management  plan  spanning  5.5  years. 

Joint  Countermine.  We  did  not  consider  the  Joint  Countermine  appropriate 
for  the  ACTD  program  because  it  was  not  technically  mature  and  did  not  meet  a 
critical  military  need  when  we  applied  our  interpretation  of  those  selection 
criteria  to  the  ACTD. 

Mature  Technology.  The  Navy  demonstration  manager  for  the  Joint 
Countermine  established  an  independent  ACTD  Review  Committee  (the 
Committee)  in  May  1995  to  assess  the  project.  The  Committee  concluded  that 
the  hardware  for  the  Advanced  Lightweight  Influence  Sweep  System  and  the 
Magic  Lantern  (Adaption)  Advanced  Technology  Demonstration  subsystems 
supporting  the  Joint  Countermine  were  not  sufficiently  mature  for  inclusion  in 
the  ACTD.  The  Committee  recommended  deleting  those  projects.  The 
Committee  also  concluded  that  the  C4I  and  Joint  Countermine  Operational 
Simulation  efforts  were  too  ambitious,  and  the  Committee  recommended 
rigorously  downscaling  those  efforts  to  meet  only  the  basic  objectives  of  the 
ACTD.  Additionally,  the  Committee  concluded  that  more  than  50  percent  of 
the  $86  million  budgeted  was  intended  for  the  C4I  and  the  Joint  Countermine 
Operational  Simulation  efforts  and  that  the  funding  was  for  the  development  of 
two  new  acquisition  programs. 

Critical  Military  Need.  Although  the  Joint  Staff  rated  the  Joint 
Countermine  overall  as  high  potential  to  meet  the  Joint  Warfighting  Capability 
Assessment  team  evaluation  criteria  and  need,  it  did  not  address  critical  military 
need.  In  addition,  we  found  no  documentation  in  which  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  declared  that  the  project  met  a  critical 
military  need. 
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Kinetic  Energy  Boost  Phase  Intercept.  We  did  not  consider  the  Kinetic 
Energy  Boost  Phase  Intercept  technically  mature  or  meeting  a  critical  military 
need  when  we  applied  our  interpretation  of  those  selection  criteria  to  the  ACTD. 
In  addition,  it  was  not  to  be  completed  within  the  timeframe  of  2  to  4  years. 

Mature  Technology.  The  DUSD(AT)  approved  the  project  as  an  FY 
1995  ACTD.  Simulation  and  analysis  was  to  provide  technical  support  to  the 
missile  and  sensor  development.  In  addition,  a  miniaturized  version  of  the  Joint 
Tactical  Information  Distribution  System  must  be  developed  from  existing 
technologies  for  command,  control,  and  communication.  The  ACTD  planned  to 
develop  those  support  systems  during  the  ACTD  project. 

Critical  Military  Need.  The  Joint  Staff  rated  the  Kinetic  Energy  Boost 
Phase  Intercept  ACTD  overall  as  medium  potential  to  meet  the  Joint 
Warfighting  Capability  Assessment  team  evaluation  criteria  and  need.  We 
could  not  find  evidence  that  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  or  any  other  authority  within  the  Department  stated  that  the  project 
met  a  critical  military  need. 

Timeframe  (2  to  4  Years).  We  determined  whether  the  ACTD  went 
beyond  the  4  years  from  the  time  that  the  DUSD(AT)  approved  the  ACTD  until 
the  completion  of  the  both  Phase  I  and  II  demonstration.  The  DUSD(AT) 
approved  the  program  in  August  1994.  The  draft  management  plan  shows  the 
program  demonstration  beginning  in  early  2000,  which  spans  almost  6  years. 

Low  Life-Cycle  Cost  Helicopter.  The  Low  Life-Cycle  Cost  Helicopter  (the 
Helicopter)  was  not  appropriate  for  the  ACTD  program  because  the  project  did 
not  meet  a  critical  military  need  when  we  applied  our  interpretation  of  that 
selection  criterion  to  the  ACTD,  and  the  Joint  Staff  rated  the  Helicopter  as  low 
potential. 

We  did  not  identify  documentation  in  which  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  stated  that  the  project  met  a  critical  military  need. 
The  only  need  identified  was  the  need  to  determine  the  cost-effectiveness  of 
leasing  commercial  helicopters  to  replace  military  helicopters.  Also,  the  Joint 
Staff  rated  the  Helicopter  overall  as  low  potential  to  meet  the  Joint  Warfighting 
Capability  Assessment  team  evaluation  criteria  and  need.  The  Helicopter 
project  resulted  from  a  contractor's  proposal  to  provide  commercial  helicopters 
to  meet  the  vertical  lift  requirement  of  the  Navy  because  of  the  aging  fleet  of 
Navy  helicopters. 

A  DUSD(AT)  official  stated  that  DUSD(AT)  supplied  funding  to  the  project  to 
overcome  resistance  by  officials  within  the  Navy.  The  DUSD(AT)  approved 
the  project  in  July  1995  and  provided  $1  million  of  DoD  funding  to  the  Navy  to 
demonstrate  the  concept  of  a  lease  charter  approach  in  place  of  procurement. 
However,  that  approach  did  not  meet  the  criteria  for  DoD  funding,  which  is  to 
fill  funding  gaps  when  integrating  several  technology  programs,  to  supply 
multiple  copies  of  the  ACTD  hardware,  or  to  support  the  2-year  residual 
period.  The  Navy  did  not  prepare  a  management  plan.  In  addition,  the  Navy 
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July  1995  analysis  of  the  cost-effectiveness  of  leasing  commercial  helicopters 
instead  of  procuring  new  helicopters  concluded  that  leasing  was  more  cost- 
effective. 

Because  the  Navy  demonstration  program  office  did  not  initially  evaluate  the 
project's  stated  objectives  to  determine  the  cost-effectiveness  of  leasing,  the 
Navy  planned  a  $20  million,  6-month,  non-ACTD  follow-on  demonstration  to 
evaluate  those  same  objectives. 

Rapid  Force  Projection  Initiative.  We  did  not  consider  the  Rapid  Force 
Projection  Initiative  technically  mature  or  meeting  a  critical  military  need  when 
we  applied  our  interpretation  of  those  selection  criteria  to  the  ACTD. 

Mature  Technology.  The  Rapid  Force  Projection  Initiative  planned  to 
use  simulations  and  analyses  to  support  the  determination  of  value  added  of 
proposed  technologies  for  the  ACTD  because  the  supporting  systems  were 
immature.  In  addition,  the  communication  architecture  used  in  the  ACTD  was 
based  on  the  evolving  battle  command  system  that  the  Army  was  developing. 

Critical  Military  Need.  The  Joint  Staff  rated  the  Rapid  Force 
Projection  Initiative  ACTD  overall  as  low  potential  to  meet  the  Joint 
Warfighting  Capability  Assessment  team  evaluation  criteria  and  need.  We 
could  not  find  evidence  in  which  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  or  any  other  authority  stated  that  the  project  met  a 
critical  military  need. 


Marginal  Projects 


In  addition  to  the  questionable  projects,  several  of  the  programs  that  we 
reviewed  did  not  meet  the  critical  military  need  selection  criterion.  We 
considered  them  marginal  candidates  because  no  authority  within  the 
Department  identified  them  as  meeting  a  critical  military  need.  The  programs 
were  the  Counter  Proliferation,  the  Cruise  Missile  Defense,  and  the 
Precision/Rapid  Counter  Multiple  Rocket  Launch. 

Critical  Military  Need.  We  could  not  find  evidence  that  the  Under  Secretary 
of  Defense  for  Acquisition  and  Technology  or  any  other  authority  stated  that  the 
projects  met  a  critical  military  need.  However,  the  Joint  Staff  reviewed  and 
rated  each  of  those  ACTDs  as  described  below. 

Counter  Proliferation.  The  Joint  Staff  reviewed  Counter  Proliferation 
as  part  of  its  review  of  the  FY  1996  ACTD  candidates.  In  a  memorandum  to 
the  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  May  15,  1995, 
the  Joint  Staff  stated  that  it  was  generally  supportive  of  the  proposed  candidates. 
However,  each  Service  representative  provided  input  to  the  overall  decision. 
Some  said  that  they  supported  the  ACTD;  others  said  that  the  projects  were 
promising  but  needed  clarification;  and  still  others  said  that  the  projects  were 
not  relevant  to  that  Service's  responsibilities  in  joint  warfare. 
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Cruise  Missile  Defense,  Precision/Rapid  Counter  Multiple  Rocket 
Launch.  The  Joint  Staff  reviewed  the  ACTDs  as  part  of  the  FY  1995 
candidates.  It  rated  the  Cruise  Missile  Defense  as  high  potential  and  the 
Precision/Rapid  Counter  Multiple  Rocket  Launch  as  medium  potential  to  meet 
the  Joint  Warfighting  Capability  Assessment  team  evaluation  criteria  and  need. 
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The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  provided 
specific  comments  to  the  executive  summary,  Part  I  -  Audit  Results,  and  Part  II 
-  Additional  Information.  He  addressed  the  selection  criteria  of  critical  military 
need,  mature  technology,  user  involvement,  and  the  2-  to  4-year  timeframe. 
Below  we  discuss  management  comments  and  provide  our  audit  response. 

Comments  on  Critical  Military  Need.  The  Under  Secretary  stated  that  our 
report  incorrectly  implies  that  a  formal  determination  by  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  is  the  appropriate  basis  for  establishing 
military  need.  He  stated  that  the  process  of  identifying  a  critical  military  need 
is  articulated  in  the  1995  ACTD  management  plan  in  which  the  warfighting 
community  identifies  the  need,  which  is  then  confirmed  by  the  Joint 
Requirements  Oversight  Council.  The  process  is  consistent  with  the  DoD 
Directive  5000.1,  "Defense  Acquisition,"  which  governs  acquisition  programs. 
The  Under  Secretary  further  stated  that  the  report's  interpretation  is  inconsistent 
with  the  actual  process  already  in  place  to  determine  military  need.  He  stated, 
however,  that  he  had  not  been  totally  consistent  in  the  documentation  of  the 
process  for  establishing  military  need  and  that  the  working  group  will  address 
the  criteria. 

Audit  Response.  We  agree  with  the  Under  Secretary's  outline  of  the  process 
identified  in  the  1995  ACTD  management  plan.  Appendix  C  of  the  report 
outlines  the  process  described  by  the  Under  Secretary.  The  process  included  the 
user's  identification  of  military  need  and  the  Joint  Staff's  review  and 
prioritization.  We  have  not  questioned  the  need  for  the  projects.  We  have 
questioned  whether  the  projects  selected  are  the  most  critical.  The  Joint  Staff 
prioritizes  the  ACTDs;  however,  the  criteria  for  the  prioritization  as  described 
in  Appendix  C  does  not  include  "critical  military  need,"  used  by  the  Breakfast 
Club  to  recommend  candidates  to  DUSD(AT)  for  selection.  For  example,  the 
Joint  Staff  rated  two  of  the  nine  projects  we  reviewed  as  low,  yet  DUSD(AT) 
approved  the  projects  for  the  ACTD  program.  Those  projects  were  the  Rapid 
Force  Projection  Initiative  and  the  Low  Life-Cycle  Cost  Helicopter. 
DUSD(AT)  has  not  indicated  the  organization  that  may  declare  that  a  proposed 
project  meets  a  critical  military  need.  Because  the  Under  Secretary  made  a 
declaration  of  military  need  for  the  Medium  Altitude  Endurance-UAV,  we  used 
his  determination  as  a  basis  for  our  assessment.  That  program  was  the  only 
ACTD  for  which  a  DoD  official  made  a  formal  declaration  of  military  need. 
The  working  group  should  address  who  may  declare  a  critical  military  need 
when  convened. 

Comments  on  Mature  Technology.  The  Under  Secretary  stated  that  the  draft 
report  misstates  the  technology  standard  as  mature  technology  when  in  fact  the 
precise  definition  is  "technology  should  be  sufficiently  mature."  He  stated  that 
the  most  senior  technologists  in  the  Department  of  Defense  reviewed  the 
approved  ACTDs  and  judged  them  employing  sufficiently  mature  technology 
for  the  program.  The  Under  Secretary  indicated  that  the  objective  of  the 


31 


Appendix  F.  Audit  Response  to  Specific  Management  Comments 


program  is  to  select  technologies  that  will  be  available  when  needed  and  not  to 
restrict  the  choice  to  those  technologies  that  are  fully  demonstrated.  He  stated 
that  the  level  of  technology  must  be  sufficient  so  that  the  residual  capability  that 
remains  with  the  user  at  the  conclusion  of  the  ACT'D  is  useful  in  an  operation 
environment. 

The  Under  Secretary  also  stated  that  the  audit  report  cites  models  and 
simulations  as  an  indication  of  the  use  of  immature  technologies  in  an  ACTD. 
He  stated  that  on  the  contrary,  models  and  simulations  are  vital  to  ACTDs 
because  the  program  manager  uses  models  and  simulations  to  plan  experiments, 
to  develop  concepts  of  operations,  and  to  supplement  knowledge  gained  from 
testing. 

Further,  the  Under  Secretary  stated  that  the  report  does  not  distinguish  between 
risks  resulting  from  integration  and  risks  resulting  from  immature  technology. 
Most  ACTDs  involve  integration  of  elements  that  require  development  of  new 
software,  which  is  not  a  technology  maturity  issue  but  rather  a  risk  issue.  The 
Under  Secretary  cited  as  an  example  the  crash  of  the  Darkstar.  He  stated  that 
the  Darkstar  crashed  because  of  deficiencies  in  modeling  and  simulation  and  not 
flight  control  software.  The  Under  Secretary  stated  that  those  issues  were 
integration  issues  and  not  issues  of  maturity. 

Audit  Response.  We  identified  inconsistent  published  guidance  concerning  the 
maturity  of  technology.  "The  Guidance  for  ACTD  Management  Plan,"  January 
1995,  uses  the  term  "maturing  technology."  The  "ACTD  Initiation  and 
Approval  Process,"  July  1995,  and  the  DoD  Directive  5000.2-R,  "Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS)  Acquisition,"  use  the  term  "mature 
technology. "  The  Under  Secretary  is  now  saying  that  the  criteria  is  "sufficiently 
mature  technology."  The  Under  Secretary  must  clarify  what  the  criteria  is  and 
must  explain  its  meaning,  whether  "mature  technology,"  "maturing 
technology,"  or  "sufficiently  mature  technology,"  to  assist  the  Services  or  other 
interested  parties  in  assessing  and  proposing  candidate  projects  for  the  ACTD 
program. 

We  agree  that  the  use  of  models  and  simulations  are  vital  to  an  ACTD  or  other 
programs  because  the  program  manager  may  use  them  to  plan  experiments,  to 
develop  concepts  of  operations,  or  to  supplement  knowledge  gained  from 
testing.  However,  the  examples  cited  in  the  report  show  that  program  office 
officials  used  models  and  simulations  to  represent  technology  not  yet  developed 
or  to  represent  software  programs  that  support  ACTDs  for  which  software 
development  was  concurrent  with  the  development  of  the  program  that  supports 
the  ACTD.  If  the  technology  is  not  yet  developed,  the  outcome  of  the  modeling 
and  simulation  is  highly  questionable.  For  example,  the  contractor  was 
developing,  concurrent  with  the  hardware  models  and  simulations  used  to 
represent  prototype,  High  Altitude  Endurance-UAV  hardware.  Additionally, 
developers  of  the  prototype  High  Altitude  Endurance-UAV  hardware  relied 
significantly  on  the  immature  models  and  simulations.  As  stated  in  the  draft 
audit  report,  the  Cost  Analysis  Improvement  Group's  assessment  of  the  Darkstar 
identified  substantial  risk  of  cost  growth  for  software  development,  not 
integration  as  claimed  by  the  Under  Secretary.  Also,  the  report  states  that  the 
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safety  investigation  results  concluded  that  the  High  Altitude  Endurance-UAV 
crashed  as  a  result  of  the  system's  flight  control  software  and  deficiencies  in  the 
system’s  modeling  and  simulation.  A  memorandum  from  the  Deputy  Program 
Director,  July  10,  1996,  confirmed  that  the  flight  control  software  was  the  cause 
of  the  Darkstar  crash.  The  issue  was  a  maturity  issue  foremost  and  secondly  an 
integration  issue.  The  issues  are  not  mutually  exclusive  but  interrelated.  DoD 
officials  must  carefully  evaluate  both  issues  in  the  selection  of  an  ACTD 
candidate.  However,  the  first  step  in  the  process  is  having  a  clear 
understanding  of  what  is  acceptable,  mature  or  maturing  technology.  This 
understanding  maybe  in  the  form  of  examples.  Those  definitions  will  help 
decision  makers  in  the  identification  of  potential  candidates  for  the  ACTD 
program  and  in  the  approval  of  projects  for  the  program. 

Comments  on  User  Involvement.  The  Under  Secretary  stated  that  the  report 
incorrectly  bases  its  interpretation  on  whether  the  user  was  involved  in  the 
ACTD  process  based  on  the  user  signing  the  management  plan,  the 
implementation  directive,  or  the  memorandum  of  agreement.  He  stated  that  the 
user  was  committed  to  support  the  Medium  Altitude  Endurance-UAV  and  the 
High  Altitude  Endurance-UAV.  The  user  made  those  commitments  during 
discussions  with  the  DUSD(AT).  He  stated  that  the  level  of  user  involvement 
during  the  ACTD  indicates  the  strength  of  the  commitment. 

Audit  Response.  We  were  unable  to  determine  the  commitment  that  the  users 
provided  to  the  programs  if  they  did  not  formally  sign  up  through  the  program 
documentation  requirements.  We  commend  the  DUSD(AT)  in  his  attempts  to 
have  the  user  signed  on  early  in  the  life  of  the  ACTD  programs.  However,  for 
the  High  Altitude  Endurance-UAV,  the  user  still  did  not  sign  the  High  Altitude 
Endurance-UAV  Management  Plan  as  of  August  1996,  2  years  after  the  ACTD 
was  initially  approved.  As  stated  in  the  audit  report,  officials  did  not  approve 
the  Management  Plan  because  of  the  definition  of  the  user's  role  as  the  user 
representative  and  refinement  of  transition  planning  issues.  Thus,  even  though 
the  user  may  have  been  informally  committed  to  the  High  Altitude  Endurance- 
UAV  ACTD,  the  user  was  not  clearly  in  agreement  on  all  program  issues  and 
most  definitely  was  not  formally  committed  to  the  program. 

While  the  user  may  have  been  involved  during  the  duration  of  the  ACTD,  the 
purpose  of  having  the  users  sign  the  management  plan,  implementation 
directive,  or  the  memorandum  of  agreement  is  to  ensure  continual  support 
throughout  the  duration  of  the  program.  Without  that  commitment,  the  users 
are  giving  little  more  than  lip  service  to  the  commitment,  and  it  falls  far  short  of 
the  intent  of  the  ACTD,  which  is  to  have  the  user  intimately  involved  from  the 
beginning  of  the  ACTD  until  completion. 

Comments  on  the  2-  to  4-Year  Timeframe.  The  Under  Secretary  stated  that 
we  had  changed  the  timeframe  from  "typically  be  2  to  4  years,"  to  a  more 
stringent  timeframe  of  "2  to  4  years."  He  indicated  that  the  timeframe  is  listed 
as  a  selection  criterion  to  distinguish  the  ACTD  process  from  the  timeframe  of 
12  years  or  more,  which  is  typically  required  of  a  formal  acquisition  program. 
He  stated  that  the  timeframe  was  intended  to  be  a  coarse  measure  and  not  a  fine 
measure,  as  the  report  interpreted. 
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Audit  Response.  We  used  a  finite  measure  to  determine  whether  the  programs 
were  to  be  completed  within  a  short  time,  as  spelled  out  in  the  ACTD  master  % 
plan.  If  the  need  is  identified  by  the  user  and  accepted  as  a  critical  military 
need,  then  the  duration  of  the  program  should  be  within  a  reasonable  timeframe 
to  meet  the  need.  Otherwise,  the  urgency  and  the  subsequent  selection  of  the 
project  for  the  ACTD  program  is  questionable.  The  High  Altitude  Endurance- 
UAV  spans  more  than  5.5  years,  and  the  Kinetic  Energy  Boost  Phase  Intercept 
program  spans  more  than  6  years.  While  the  duration  of  the  programs  does  not 
significantly  exceed  the  selection  criteria  goals,  they  are  indicators  of  the 
systemic  problem  that  the  overall  ACTD  programs  face.  The  problems  concern 
the  vagueness  of  the  selection  criteria,  which  is  the  overall  theme  of  the  audit 
report.  The  ambiguity  in  definition  and  terms  will  only  lead  critics  of  the 
program  to  conclude  that  the  ACTD  program  is  circumventing  the  acquisition 
process  and  not  accelerating  the  acquisition  process  through  the  use  of  mature 
technology  to  meet  the  critical  military  needs  of  the  warfighting  community. 

Comments  on  the  Low  Life-Cycle  Cost  Helicopter.  The  Under  Secretary 
stated  that  nothing  in  the  ACTD  solution  criteria  precludes  projects  that  evaluate 
commercial  solutions  to  military  needs.  The  Under  Secretary  stated  that  the 
ACTD  program  provides  an  excellent  vehicle  for  assessing  military  utility  of 
such  solutions. 

Audit  Response.  The  Low  Life-Cycle  Cost  Helicopter  project  is  to  determine 
whether  commercial  helicopters  can  be  used  instead  of  procuring  new  military 
helicopters  to  lower  operating  cost.  While  we  believe  that  such  projects  have 
merit,  we  do  not  believe  that  they  meet  the  intent  of  the  ACTD  program,  which 
is  to  quickly  bring  advanced  technology  to  the  warfighter  to  meet  a  critical 
military  need.  The  use  of  the  ACTD  program  for  projects  such  as  the  Low 
Life-Cycle  Cost  Helicopter  could  jeopardize  the  ACTD  program  by  causing 
concern  that  the  ACTD  program  will  be  used  to  circumvent  other  means  to  test 
such  concepts. 
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Under  Secretary  of  Defense  for  Acquisition  and 
Technology  Comments 


ACQUISITION  AND 
TECHNOLOGY 


THE  UNDER  SECRETARY  OF  DEFENSE 
3010  DEFENSE  PENTAGON 
WASHINGTON.  D.C.  20301-3010 


FEB  1  8  1997 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

Subject:  Audit  Report  on  Advanced  Concept  Technology 

Demonstrations  (ACTDs)  (Project  No.  6AB-0018) , 
memo  dated  December  9,  1996 

Thank  you  for  the  opportunity  to  review  the  findings 
summarized  in  your  draft  report.  I  appreciate  your  efforts  to 
evaluate  the  effectiveness  of  the  ACTD  program,  however,  I 
have  serious  concerns  with  your  methodology  and  the 
conclusions  you  have  reached;  thus,  I  do  not  believe  that  the 
draft  report  should  stand  without  substantial  revision. 

The  two  most  serious  concerns  with  the  draft  audit 
report  have  to  do  with  your  assessment  of  the  ACTD  selection 
criteria.  First,  your  report  implies  that  a  formal  USD(AtT) 
determination  is  the  appropriate  basis  for  establishing 
military  need.  This  is  not  correct.  Second,  the  draft 
report  misstates  our  technology  standard  as  'mature 
technology*  when  in  fact  the  precise  criterion  is 
-technology  should  be  sufficiently  mature.*  Several  errors 
flow  from  this  misstatement.  These  two  issues  are  described 
below  and  a  more  complete  discussion  of  the  concerns  is 
contained  in  the  attachment. 

Military  needs  for  ACTDs  are  first  articulated  by  the 
warfighter  community  and  then  confirmed  by  the  Joint 
Requirements  Oversight  Council  (JROC) .  This  process  is 
described  in  the  1995  ACTD  Master  Plan.  It  is  patterned 
after  the  process  used  to  establish  needs  in  the  formal 
acquisition  process  and  is  consistent  with  DODD  5000.1.  All 
ACTDs  approved  to  date  have  followed  this  process,  including 
a  review  by  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff 
(VCJCS)  or  the  full  Joint  Requirements  Oversight  Council 
(JROC) ,  who  have  confirmed  that  these  are  appropriate 
projects  for  ACTDs.  The  auditors*  interpretation  of  the 
process  for  establishing  military  needs  is  inconsistent  with 
the  published  process  and  with  Department  policy  regarding 
responsibility  for  needs  prioritization.  It  should  not  be 
used  as  the  basis  for  an  assessment  of  the  ACTD  selection 
process . 

The  second  concern  is  your  statement  that  four  of  the 
ACTDs  did  not  use  mature  technology.  All  of  these  ACTDs  were 
reviewed  by  the  user  and  by  the  Deputy  Under  Secretary  of 
Defense  (Advanced  Technology)  (DQSD(AT) )  Breakfast  Club, 
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which  is  comprised  of  the  most  senior  technologists  in  the 
Department  of  Defense.  These  ACTDs  were  judged  to  employ 
sufficiently  mature  technologies,  which  is  the  precise 
selection  criterion,  not  'mature  technologies'  as  used  in  the 
IG's  evaluation  process.  While  this  may  appear  to  be  a 
subtle  distinction,  it  led  to  improper  measures  and 
interpretations  being  applied  during  the  audit  process.  For 
example,  models  and  simulations  (M&S)  were  cited  as  an 
indicator  of  the  use  of  immature  technologies  in  an  ACTD.  To 
the  contrary,  M&S  are  a  vital  element  of  ACTDs.  They  are 
used  to  plan  experiments,  develop  CONOPS  and  to  supplement 
knowledge  gained  from  field  testing  of  the  core  elements  of 
the  ACTD.  They  are  not  used  in  lieu  of  field  testing  of  core 
elements.  While  conceptual  systems  are  sometimes  represented 
in  the  M&S,  they  are  there  to  assess  growth  beyond  the  core 
capability  of  the  ACTD.  This  should  not  have  been 
interpreted  as  an  indication  that  the  core  systems  are  not 
sufficiently  mature.  Furthermore,  the  report  does  not 
distinguish  between  risks  due  to  integration  of  complex 
systems  and  risks  due  to  immature  technology.  It  is 
important  to  make  this  distinction  since  integration  risks 
are  addressed  under  a  separate  selection  criterion.  The 
technology  maturity  in  the  nine  ACTDs  that  are  addressed  in 
your  report  is  consistent  with  the  intent  of  the  ACTD 
selection  criteria. 

It  is  clear  that  the  ACTD  selection  criteria  has  not 
been  interpreted  correctly  by  the  audit  team  and  others  may 
encounter  similar  problems.  I  therefore  agree  with  your 
recommendation  to  assemble  a  working  group  to  clarify  the 
intent  of  the  criteria.  We  will  invite  a  representative  of 
your  organization  to  participate  in  that  effort.  We  will  also 
re-examine  the  ACTDs  selected  in  FY97  to  ensure  they  are 
consistent  with  the  intent  of  the  selection  criteria.  I 
would  urge  that  the  assessment  found  in  Part  I,  Audit 
Results,  pages  3  through  5,  and  Appendix  E  be  removed  from 
the  final  report.  The  focus  of  the  report  should  be  on  the 
need  for  clarification  of  selection  criteria.  I  would  be  glad 
to  provide  any  assistance  needed  in  that  regard. 


Attachment 


Paul  G.  Kaminski 
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Comments  on  the  DoD  Inspector  General  Draft  Audit  Report 
on 

Advanced  Concept  Technology  Demonstrations 


Ixeeutive  Summary 

1.  Page  i,  paragraph  4,  regarding  audit  results. 

Comment :  The  statement  that  five  of  the  ACTDs  were 
questionable  is  based  on  incorrect  interpretations  of  the 
ACTD  selection  criteria  regarding  military  need  and 
technology  maturity.  The  IG's  assessment  of  military  need 
was  based  on  whether  or  not  a  critical  need  had  been 
documented  by  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  (USD[A&TJ).  This  is  not  an  appropriate  basis 
for  judging  military  need,  nor  is  it  consistent  with  the 
selection  process  as  described  by  the  Deputy  Under  Secretary 
of  Defense  (Advanced  Technology)  (DUSD(ATJ)  in  the  1995  ACTD 
Master  Plain.  The  plan  indicates  that  the  need  for  an  ACTD  is 
first  articulated  by  the  warfighter  community  and  then 
confirmed  by  the  Joint  Requirements  Oversight  Council  (JROC)  . 
This  approach  is  patterned  after  the  process  used  to 
establish  needs  in  the  formal  acquisition  process.  DODD 
5000.1  assigns  the  Joint  Requirements  Oversight  Council 
(JROC)  responsibility  for  the  prioritization  of  military 
needs.  All  ACTDs  approved  to  date  have  followed  the  process 
described  in  the  ACTD  Master  Plan  and  have  been  confirmed  by 
the  chairman  of  the  JROC  to  be  appropriate  projects  for 
ACTDs . 

The  criterion  which  concerns  technology  maturity  states  that 
"technology  should  be  sufficiently  mature.*  The  report 
revised  this  criterion  to  be  "mature  technology*  which  in  not 
consistent  with  the  intent.  The  objective  is  to  select 
technologies  that  will  be  available  when  needed,  not  to 
restrict  the  choice  to  those  technologies  that  have  already 
been  fully  demonstrated. 

All  nine  of  the  ACTDs  addressed  in  the  report  were  reviewed 
under  the  established  process  and  judged  by  competent 
authority  to  address  a  critical  military  need  and  to  employ 
sufficiently  mature  technology.  This  paragraph  should  be 
revised  to  indicate  that  the  selection  criteria  and  process 
were  observed  in  the  selection  of  these  ACTDs .  The  issue 
here  should  be  the  need  to  clarify  the  criteria  to  allow  it 
to  be  better  understood  by  those  who  are  not  directly 
involved  in  the  selection  process. 
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Fart  Z  -  Audit  Results 

1.  Page  3,  first  paragraph,  which  discusses  the  ACTD 
selection  process. 

Comment :  As  indicated  in  the  previous  section,  the  statement 
that  five  of  the  ACTDs  are  questionable  is  based  on  incorrect 
interpretations  of  the  selection  criteria.  This  also  applies 
to  the  statement  that  the  selection  criteria  has  been 
" inconsistently  applied.*  When  the  selection  process  was 
audited  to  see  if  the  criteria  had  been  applied  properly,  the 
criteria  was  modified  in  a  significant  way.  The  criterion 
"sufficiently  mature  technology'  was  changed  to  "mature 

technology'  and  the  criterion  "timeframe - should  typically 

be  2-4  years'  was  changed  to  a  more  rigid  "2-4  year 
timeframe.'  Furthermore,  very  specific  tests  for  compliance 
(e.g.,  use  of  models  &  simulation  C4I  development  as 
indicators  of  immaturity,  USD(A&T)  documentation  to  assess 
military  need,  signature  on  plans  or  agreements  to  assess 
user  involvement)  were  then  applied.  Any  apparent 
inconsistencies  are  the  result  of  these  narrow 
interpretations  and  tests  for  compliance.  The  wording  we 
have  chosen  for  the  ACTD  selection  criteria  reflects  our 
intent  to  consider  a  wide  range  of  ACTD  candidates  and  to 
choose  those  that  have  the  highest  potential  payoff.  The 
criteria  have  been  applied  in  accordance  with  this  intent  and 
in  a  consistent  manner.  The  report  should  be  revised  to 
reflect  this. 

The  statement  that  only  one  of  the  22  projects  selected  as 
ACTDs  in  FY95  and  FY96  was  proposed  by  a  Military  Department 
is  incorrect .  The  report  should  be  revised  to  indicate  that 
seven  of  those  projects  were  proposed  by  either  the  Army,  the 
Navy  or  both.  The  projects  proposed  by  the  Army  were 
Precision  Rapid  Counter  Multiple  Rocket  Launcher,  Rapid  Force 
Projection  Initiative,  and  Joint  Logistics  and  Joint 
Countermine.  The  Navy  proposed  Precision  SIGINT  Targeting 
System,  Low  Life  Cycle  Cost  Helicopter,  Cruise  Missile 
Defense  (Phase  I),  and  Joint  Countermine.  Given  the  emphasis 
on  joint  capabilities  and  the  strong  role  in  the  ACTD  process 
of  the  Joint  Staff,  the  Unified  CINCs,  DARPA,  and  Joint 
Project  Offices,  six  proposals  originating  with  the  Military 
Departments  is  considered  a  reasonable  balance. 

2.  Page  6,  concerning  technology  maturity. 

Comment ;  The  statement  that  four  of  the  ACTDs  did  not  use 
mature  technology  highlights  the  use  in  the  report  of 
"mature'  instead  of  "sufficiently  mature'  to  describe 
acceptable  technology.  The  criterion  was  selected  to 
recognize  that  there  are  two  sides  to  the  maturity  coin.  On 
one  side  of  the  coin,  we  do  not  want  to  restrict  the  choice 
to  those  technologies  that  have  already  been  fully 
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demonstrated.  We  need  to  be  able  to  select  evolving 
technologies,  as  long  as  they  will  be  available  when  needed. 
This  is  a  significant  point  and  can  allow  use  of  technologies 
that  are  as  much  as  2-3  years  newer  than  otherwise  possible. 

A  good  example  of  this  is  the  computer  industry  which  is 
introducing  new  generations  of  equipment  every  eighteen 
months.  In  such  cases,  it  is  wise  to  allow  technology 
development  work  to  be  performed  within  or  concurrent  with  an 
ACTD  if  there  is  reasonable  confidence  in  the  time  required 
to  complete  development.  The  criterion  *  sufficiently  mature 
technology*  also  allows  the  use  of  less  mature  technologies 
when  they  operate  in  parallel,  such  that  overall  risk  is 
acceptable . 

On  the  other  side  of  the  coin,  the  level  of  technology  must 
be  sufficient  so  that  the  residual  capability  that  is  left 
with  the  user  at  the  conclusion  of  the  ACTD  is  useful  in  an 
operational  environment.  This  means  that  not  only  must  the 
performance  meet  the  needs,  but  the  configuration  must  also 
be  suitable  for  use  by  the  troops. 

The  draft  audit  report  tests  for  "mature  technology'  and  does 
not  consider  an  ACTD  to  use  mature  technology  if: 

a)  the  project  used  models  and  simulation  to  represent 
technology  not  yet  developed;  or 

b)  software  programs  such  as  C4I,  were  developed 
concurrently  with  other  programs  that  support  the 
ACTD 

Both  the  interpretations  and  the  way  in  which  they  were 
applied  have  led  to  erroneous  conclusions  as  described  below: 

Modeling  and  Simulation (M&S) 

Four  ACTDs  were  cited  as  relying  on  modeling  and  simulation 
because  supporting  programs  were  not  mature.  It  is  important 
to  point  out  that  M&S  is  a  valuable  tool  for  use  in 
development,  integration,  planning,  CONOPs  development,  and 
assessment  of  utility,  M&S  is  particularly  important  for 
ACTDs  which  involve  systems -of -systems.  These  types  of  ACTDs 
generally  involve  large  military  exercises.  To  plan  these 
exercises  requires  reasonably  good  insight  into  the 
performance  and  CONOPs  of  the  forces  involved  and  into  the 
associated  interactions.  For  systems-of-systems,  it  is 
impractical  to  do  that  solely  through  mission  rehearsal  using 
live  forces .  M&S  provides  an  effective,  low  cost 
alternative.  In  addition,  the  assessment  of  military  utility 
for  an  ACTD  must  often  consider  the  implications  of  the 
proposed  capability  on  the  extended  battlefield  -  not  ]ust 
between  a  few  opposing  elements.  M&S  is  the  proper  way  to 
make  this  assessment.  For  these  and  other  reasons,  use  of 
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M&S  throughout  the  development  process  is  strongly  encouraged 
by  DoD  5000. 2R  (Section  3.44). 

In  a  number  of  ACTDs,  M&S  has  also  been  used  to  examine 
conceptual  systems  that  are  not  a  part  of  the  ACTD 
capability.  These  conceptual  systems  frequently  are  not  # 
mature,  the  components  will  not  be  physically  present  during 
the  ACTD,  and  these  capabilities  will  not  be  left  with  the 
user  as  residuals.  For  these  reasons  they  are  not  part  of 
the  core  capability  and  they  are  examined  on  a  non- 
interference  basis  to  provide  insight  into  issues  related  to 
future  evolution  of  the  ACTD  core  capabilities.  This  is  an 
efficient  and  inexpensive  way  to  get  added  value  from  the 
Actd  The  iiroortant  point  is  that  core  capabilities  always 
exist  as  real  hardware  and  software;  M&S  is  not  being  used  in 
lieu  of  actual  testing  of  those  capabilities.  The  report 
should  be  revised  to  remove  statements  that  M&S  is  being  used 
because  technology  is  not  mature. 


Three  ACTDs  were  cited  as  relying  significantly  on  software 
or  C4I  development  and  integration  that  was  not  based  on 
mature  technology.  However,  most  of  the  ACTDs  involve  the 
integration  of  elements  or  capabilities  that  require  > 

development  of  new  software.  Software  development  per  se  is 
not  a  technology  maturity  issue.  The  software  development 
process  is  well  established  and  predictable  if  the  underlying 
processes  and  behaviors  are  well  defined.  Even  where  there 
are  a  significant  number  of  C4I  links  involved,  it  is 
incorrect  to  conclude  that  the  number  in  itself  implies 
immature  technology.  A  case  in  point  is  the  Precision  Rapid 
Counter  Multiple  Rocket  Launcher  (PRC-MRL)  ACTD.  This  ACTD 
involved  linkage  of  not  single  service,  but  multi-service 
sensors  and  shooters.  Even  so,  the  PRC-MRL  ACTD  was  recently 
completed  and  has  been  judged  to  have  been  highly  successful. 
The  final  demonstration  occurred  with  a  major  exercise  m 
Korea  and  included  fully  exercising  the  system  linkage.  The 
report  should  be  revised  to  remove  this  basis  for  assessing 
technology  maturity. 

The  draft  audit  report  appears  to  consider  risk  due  to 
immature  technology  and  risk  due  to  integration  activities  to 
be  synonymous  and  they  are  not.  A  program  which  uses  nothing 
but  mature  technologies  can  still  incur  significant  technical 
risk  due  to  the  complexity  of  the  integration  effort  . 
required,  or  perhaps  due  to  the  use  of  mature  commercial 
technologies  in  a  military  environment.  For  example, 
building  a  complex  computer  network  entirely  from  mature, 
commercially  available  components  does  not  ensure  that . the 
network  integration  will  proceed  smoothly  or  that  it  will 
operate  flawlessly.  There  are  system  design,  integration, 
and  operations  risks  that  must  be  considered.  There  are  two 
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separate  selection  criteria  that  relate  to  risk  in  an  ACTD. 
The  first  is  the  criterion  that  addresses  technology 
maturity.  The  second  is  the  criterion  that  relates  to 
identifying,  understanding  and  accepting  other  risks  that  are 
inherent  in  ACTDs,  such  as  integration  and  programmatic 
risks.  The  risks  due  to  the  complexity  of  integrating 
sensors  and  shooters  should  be  addressed  in  the  criterion  for 
overall  risk  rather  than  as  a  technology  maturity  issue. 


Page  6 


3.  Page  6,  concerning  critical  military  need. 

PnnHTiRnt:  The  report  states  that  it  was  necessary  to  make  an 

interpretation  of  this  requirement  because  'the  Deputy  Under 
Secretary  of  Defense  (Advanced  Technology)  (DUSD t AT] )  did  not 
define  what  constitutes  an  urgent  military  need  or  who  may 
declare  the  urgent  need  for  ACTD  candidates.'  The  report 
interpreted  this  to  mean  that  a  written  determination  of 
critical  need  must  have  been  issued  by  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  (USD[A&T])  in  order  to 
satisfy  this  criterion.  As  stated  earlier,  this 
interpretation  is  inconsistent  with  the  process  for  assessing 
military  need  that  was  published  by  the  DUSD(AT)  in  the 
Advanced  Technology  Concept  Demonstration  Master  Plan  dated 
April  1995.  It  is  also  inconsistent  with  DoD  5000.1  which 
assigns  the  JROC  the  responsibility  for  establishing 
priorities  for  military  needs.  The  determination  of  military 
needs  for  ACTDs  is  made  by  the  JKOC.  Furthermore,  in  the 
final  balancing  of  ACTD  candidates,  military  needs,  and 
available  resources,  the  DUSD  (AT)  consulted  with  USD  (A&T) 
and  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  (V£JCS)  to 
obtain  their  concurrence  before  final  approval  of  both  the 
FY95  and  FY96  ACTDs.  The  audit  report  should  be  revised  to 
reflect  the  established  process  for  assessing  military  need 
and  to  reflect  that  military  need  for  each  of  and 

FY96  ACTDs  was  confirmed  as  described  in  the  published 
selection  process. 

4.  Page  7,  concerning  user  involvement. 

Comment :  The  draft  of  the  audit  report  indicates  that  two  of 

the  nine  ACTDs  examined  did  not  satisfy  this  criterion.  The 
conclusion  was  based  on  an  interpretation  by  the  auditors 
that  intimate  user  involvement  should  be  determined  by 
whether  the  user  had  signed  the  Management  Plan, 
Implementation  Directive,  or  a  Memorandum  of  Agreement. 
However,  these  documents  are  normally  approved  after  the  ACTD 
has  been  approved  and  are  not  available  at  the  time  the 
selection  decisions  are  made.  At  the  time  of  selection,  CINC 
USACOM  was  committed  to  the  support  of  both  the  Medium 
Altitude  Endurance  (MAE)  Unmanned  Aerial  Vehicle  (UAV)  and 
the  High  Altitude  Endurance  (HAE)  UAV  ACTDs.  Those 
commitments  were  made  during  discussions  between  CINC  USACOM 
and  the  DUSD  (AT)  .  Prior  to  the  approval  of  each  of  these 
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ACTDs ,  USACOM  identified  an  Operations  Manager  to  be 
responsible  for  all  user  support.  The  strength  of  commitment 
is  clearly  demonstrated  by  the  significant  level  of  user 
involvement  during  the  ACTD.  For  example,  for  the  MAE  UAV 
ACTD,  the  total  involvement  in  this  ACTD  by  USACOM  and  their 
combatant  commands,  FORSCOM,  CINCLANFLT,  and  ACC  was  in 
excess  of  200  man-years.  This  includes  US  Army  soldiers  from 
the  Military  Intelligence  Battalion  (Low  Intensity)  who  were 
assigned  TDY  to  work  on  the  program.  The  statement  in  the 
audit  report  that  the  user  was  not  intimately  involved  in 
this  ACTD  is  clearly  in  error.  Since  all  of  these  ACTDs 
began  more  than  a  year  ago,  there  is  an  excellent  opportunity 
to  evaluate  the  strength  of  user  commitment  based  on  actual 
involvement,  which  is  the  central  issue  with  this  criterion. 
The  report  should  be  revised  to  remove  the  statement  that  the 
user  has  not  been  intimately  involved  in  both  of  these  ACTDs. 

5.  Page  7,  concerning  timeframe  for  ACTDs. 

Comment :  The  report  indicates  that  two  of  the  nine  ACTDs 

examined  failed  to  meet  the  2-4  year  timeframe.  However,  the 
criterion  states  "typically  2-4  years*  not  "2-4  years*  as  the 
report  indicates.  The  timeframe  is  listed  as  a  selection 
criterion  to  distinguish  the  ACTD  process  from  the  timeframe 
of  12  years  or  more  typically  required  under  the  formal 
acquisition  process.  This  is  intended  to  be  a  course 
measure,  not  a  fine  measure  as  the  auditors  have  interpreted. 
The  range  was  used  to  indicate  variation  from  ACTD  to  ACTD 
and  the  word  "typically*  was  used  to  indicate  that  these 
limits  were  to  be  applied  to  the  group  of  ACTDs  and  were  not 
firm  limits  for  an  individual  ACTD.  Of  the  two  ACTDs  in 
question,  one  was  planned  to  be  completed  in  less  than  2 
years  and  one  in  greater  than  4  years.  The  timeframes  were 
considered  to  be  consistent  with  the  scopes  of  the  ACTDs  and 
were  acceptable.  Furthermore,  if  an  ACTD  can  be  conducted  in 
less  than  2  years,  that  should  not  be  considered  to  be  a 
negative  factor.  The  report  should  be  revised  to  indicate 
that  the  ACTDs  approved  in  both  FY95  and  FY96  were  typically 
2-4  years  in  duration. 

6.  Page  7,  last  paragraph,  concerning  questioned  projects. 

Comment :  The  Low  Life-Cycle  Cost  Helicopter  ACTD  was 
questioned  because  it  "was  not  advancing  technology  to  meet 
an  military  need,  rather  this  project  was  to  demonstrate  an 
acquisition  strategy:  the  viability  of  commercial  helicopters 
to  meet  Navy  requirements.*  The  issue  here  is  whether  it  is 
appropriate  to  use  ACTDs  to  evaluate  commercial  solutions  to 
military  needs  that  have  traditionally  required  a  unique 
military  solution.  Are  they  effective  and  suitable  for  use 
in  a  military  environment?  There  is  nothing  in  the  ACTD 
solution  criteria  to  preclude  these  types  of  projects.  In 
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fact,  ACTDs  provide  an  excellent  vehicle  for  assessing  the 
military  utility  of  such  solutions. 

7.  Page  8,  concerning  Joint  Staff  comments. 

Comment ;  In  the  review  of  proposed  candidates,  the  final 
determination  of  military  need,  and  the  judgment  of  whether 
that  need  is  sufficient  to  justify  an  ACTD,  rests  with  the 
VCJCS,  who  chairs  the  JROC.  The  VCJCS  must  integrate  the 
views  within  the  JROC.  As  is  frequently  the  case  for 
projects  that  have  a  joint  flavor,  the  level  of  support 
across  the  Military  Departments  is  not  uniform  because  the 
contribution  the  capability  makes  to  all  Departments  is  not 
equal.  In  a  similar  way,  the  contribution  to  all  mission 
areas  is  not  likely  to  be  the  same  and  the  individual  JWCAs 
are  unlikely  to  reach  identical  views.  The  responsibility 
for  providing  a  fully  integrated  picture  rests  with  the 
VCJCS .  He  has  indicated  that  each  of  the  ACTDs  selected  to 
date  was  appropriate  in  that  regard. 

8.  Page  9,  first  paragraph,  concerning  dissemination  of 
selection  criteria . 

Comment :  This  paragraph  should  be  revised  to  indicate  that 

six  of  the  FY95  and  FY96  projects  were  proposed  by  either  the 
Army  or  the  Navy.  The  projects  proposed  by  the  Army  were 
Precision  Rapid  Counter  Multiple  Rocket  Launcher,  Rapid  Force 
Projection  Initiative,  and  Joint  Logistics.  The  Navy 
proposed  Precision  SIGINT  Targeting  System,  Low  Life  Cycle 
Cost  Helicopter,  and  Cruise  Missile  Defense  (Phase  I) . 

9.  Page  9,  third  paragraph,  concerning  defined  criteria  and 
requirements . 

Comment:  DoD  Regulation  5000. 2-R  applies  to  major 

acquisition  programs  and  does  not  apply  to  ACTDs.  The 
paragraph  in  that  regulation  relating  to  ACTDs  is  a  concise 
description  of  the  ACTD  process  and  was  included  in  the 
regulation  for  information  purposes.  The  purpose  was  to 
provide  an  example  of  a  nonstandard  path  by  which  a  program 
might  enter  the  formal  acquisition  process.  The  wording  was 
not  intended  to  serve  as  the  basis  for  selection  criteria  for 
the  ACTD  process.  However,  to  avoid  future  # 

misunderstandings,  the  wording  is  being  revised  in  Change  2 
to  the  document  to  indicate  that  *ACTDs  are  a  means  of 
demonstrating  the  use  of  emerging  or  mature  technology  to 
address  critical  military  needs.* 

10.  Page  9,  concerning  mature  technology  and  the  Darks  tar 
ACTD . 

Comment :  The  cause  of  the  DarkStar  mishap  on  the  second 

flight  was  not  tied  to  issues  of  technology  maturity  but 
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rather  issues  relating  to  the  integration  of  the  technologies 
in  the  DarkStar  system.  The  systems  flight  control  software 
supported  DarkStar  through  the  first  fully  autonomous  flight 
of  the  UAV.  The  improved  software  and  other  changes 
incorporated  in  the  second  air  vehicle  will  correct  the 
wheelbarrowing  seen  on  the  second  flight  ^ 
flight  envelope.  Modeling  and  simulation  fidelity  has  been 
increased  and  the  lessons  learned  have  been  applied  to  reduce 
flight  test  risk.  The  risks  associated  with  development  of 
the  DarkStar  were  identified  and  accepted  at  the  time  the 
ACTD  was  approved. 

11.  Page  10,  concerning  inconsistent  terminology. 

comment:  The  comment  regarding  the  need  to  be  consistent  in 

the  terminology  used  in  various  documents  concerning  ACTDs  is 
valid  and  standard  terminology  will  be  adopted  and 
appropriate  changes  will  be  made  in  the  next  update  to  the 
Acquisition  Deskbook  and  the  ACTD  Master  Plan. 

12.  Page  10,  concerning  the  need  for  clearer  definition  of 
mature  technology. 

rnmment :  The  comment  concerning  the  need  to  clarify  the 

definition  of  mature  technology  is  also  valid  and  a  working 
group,  comprised  of  representatives  from  OSD,  the  Joint 
Staff,  developers,  users,  and  the  test  community  will  be 
formed  to  clarify  the  definition.  A  representative  of  the 
IG's  office  will  also  be  invited  to  participate. 

13.  Page  11,  concerning  military  need. 

Comment:  As  indicated  in  paragraph  11  above,  military  needs 

are  established  by  the  process  described  in  the  ACTD  Master 
Plan.  The  Chairman  of  the  JROC  indicated  that  all  of  the 
FY95  and  FY96  ACTDs  were  considered  to  be  appropriate  ACTDs . 
The  basis  used  in  the  report  to  assess  military  need  is 
incorrect.  The  report  should  be  revised  to  reflect  that 
military  need  for  all  of  the  ACTDs  was  established  in 
accordance  with  the  defined  process.  We  agree  that  we  have 
not  been  totally  consistent  in  the  documentation  of  the 
process  for  establishing  military  need  and  we  will  address 
that  subject  in  the  working  group  discussed  m  the  preceding 
comment . 


14.  Page  13,  concerning  support  for  ACTDs. 

Comment :  The  statement  that  the  Army  was  the  only  service 

that  proposed  ACTD  candidates  is  incorrect  and  should  be 
removed.  The  statement  concerning  the  complexity  of  the 
Joint  Countermine  ACTD  is  correct.  However,  there  is  a 
critical  military  need  for  this  capability  and  the 
technologies  involved  have  been  evaluated  on  three  separate 
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occasions  and  have  been  judged  to  be  sufficiently  mature  to 
support  the  ACTD  objectives.  The  ACTD  process  is  an  . 

excellent  mechanism  for  the  integration  of  this  type  of  joint 
capability.  Since  we  intend  the  process  to  be  capable  of 
conducting  demonstrations  of  this  complexity,  the  task  is  to 
make  that  clear  in  our  selection  criteria. 


15.  Page  16,  regarding  recommendations. 

Comment :  We  concur  that  the  selection  criteria  can  be 

clarified  and  as  indicated  in  paragraph  11  above,  we  will 
establish  a  team  for  that  purpose  and  we  will  publish  the 
resulting  selection  criteria  in  the  Defense  Acquisition 
Deskbook.  We  will  also  ensure  that  the  approved  FY97  ACTDs 
meet  the  resulting  criteria. 


Part  II  -  Additional  Information 

1.  Page  26,  concerning  the  Medium  Altitude  Endurance-UAV. 

pomment :  The  Unmanned  Aerial  Vehicles  (UAV)  Joint  Project 

Office  under  the  PEO  for  Cruise  Missiles  and  UAV  served  as 
the  demonstration  manager.  The  U.S.  Atlantic  Command  served 
as  the  operational  manager. 

2.  Page  27-28,  concerning  the  High  Altitude  Endurance 
Unmanned  Aerial  Vehicle  (HAE  UAV)  . 


Comment : 


Technology  Maturity 

The  technologies  used  in  this  ACTD  were  mature  technologies 
at  the  time  of  ACTD  initiation.  The  fabrication  technology 
had  been  demonstrated,  the  avionics  are  not  state  of  the  art 
systems,  the  software  development  task  is  not  large  by  DoD 
standards  (less  than  250,000  lines  of  code),  and  the 
propulsion  system  is  off  the  shelf.  It  was  the  integration 
which  was  deemed  medium  to  high  risk  within  the  time  and 
dollars  estimated  by  the  contractor  in  June  of  1994.  The 
contractors  developing  the  software  have  a  solid  track  record 
in  this  area  and  are  rated  as  high  as  SEI  level  3.  The 
flight  controls  software  was  not  developed  before  the  ACTD 
but  that  was  judged  to  be  acceptable. 


The  cause  of  the  accident  on  the  second  test  flight  was 
related  to  air  vehicle  wheelbarrowing  which  subsequently  led 
to  an  interaction  of  the  vehicle  inertia  and  the  landing  gear 
dynamics  that  was  exacerbated  by  the  main  landing  gear 
touching  the  ground.  The  flight  control  software  did  not 
play  a  role  in  the  accident.  The  program  intends  to  prevent 
the  occurrence  of  wheelbarrowing  using  the  controls  on  air 
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vehicle  #2.  Deficiencies  in  the  modeling  and  simulation  were 
identified  and  have  been  corrected.  These  are  not  technology 
maturity  issues,  but  rather  issues  relating  to  the  complex 
system  integration  task  on  this  ACTD*  The  risks  were 
identified  and  accepted  at  the  time  the  ACTD  was  approved. 

critical  Military  Need 

The  HAE  system  will  provide  the  warfighter  with  synoptic, 
real  time,  high  resolution  surveillance  of  the  battlefield. 
This  is  a  current  deficiency  which  was  identified  as  early  as 
the  mid-80's  in  the  RSTA  Mission  Need  Statement  and  was 
revalidated  by  JCS  after  Desert  Storm.  The  military 
importance  of  this  capability  to  the  warfighter  was  confirmed 
by  the  VCJCS  prior  to  approval  of  this  ACTD.  The  basis  used 
by  the  IG  to  assess  this  criterion  was  not  valid  and  the 
report  should  be  revised  to  reflect  the  correct  process. 

user  Intimately  Involved 

The  U.S.  Atlantic  Command  (USACOM)  was  fully  committed  to 
this  project  prior  to  its  approval  as  an  ACTD.  At  the  start 
of  the  ACTD,  USACOM  designated  an  Operations  Manager  to 
oversee  all  user  support  to  the  ACTD.  USACOM  had  agreed  to 
be  responsible  for  the  development  of  CONOPs,  operational 
employment  in  military  exercises,  and  the  assessment  of 
military  utility.  They  will  be  represented  in  many  of  these 
areas  by  their  Combatant  Command,  the  USAF  Air  Combat 
Command.  There  have  been  User  conferences  conducted  at 
approximately  9  month  intervals,  ground  segment  experiments 
with  military  service  participation,  user  attendance  at 
program  reviews  and  excellent  participation  of  the  User  on 
the  HAE  oversight  committee.  Full  user  involvement  in. this 
ACTD  has  never  been  issue  and  the  report  should  be  revised  to 
reflect  that  fact. 

Timeframe 

The  HAE  UAV  ACTD  was  scheduled  to  be  five  and  a  half  years  in 
length.  While  a  little  longer  than  typical,  this  was 
consistent  with  the  objective  of  this  ACTD  and  was  considered 
to  be  appropriate.  Also,  it  should  be  noted  that  the  user 
field  demos  required  a  full  two  years  of  this  period, 
significantly  greater  than  for  many  ACTDs . 

3.  Page  28-29,  concerning  Joint  Countermine. 

Comment 

Technology  Maturity 

The  issue  of  maturity  of  the  components  and  C4I  architecture 
used  in  this  ACTD  has  been  reviewed  on  three  separate 
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occasions.  Prior  to  the  approval  of  the  ACTD,  the  technology 
was  reviewed  by  the  Breakfast  Club  and  it  was  judged  to  be 
sufficiently  mature.  This  ACTD  is  a  case  in  which  duplicate 
technologies  are  being  assessed  for  some  aspects,  thus 
reducing  the  risk  due  to  technology.  Because  of  the 
complexity  of  this  ACTD,  the  Deputy  CINC  USACOM  conducted  a 
separate  review  and  he  also  concluded  that  the  technology  was 
sufficiently  mature.  In  a  1995  review  by  the  Defense  Science 
Board,  concerns  were  expressed  that  the  technology  was  too 
mature  and  that  more  aggressive  technologies  should  be 
included. 

The  IG  audit  team  also  raised  a  question  about  the  models  and 
simulation  effort  being  used  to  represent  elements  that  are 
not  physically  present  due  to  lack  of  maturity.  As  explained 
earlier,  this  is  not  the  reason  for  the  use  of  models  and 
simulation.  All  core  elements  of  this  ACTD  will  be  present 
in  one  or  more  of  the  planned  demonstrations.  In  the  case  of 
the  Joint  Countermine  Operational  Simulation,  this  capability 
is  an  element  of  the  operational  system.  It  is  vital  to  the 
real  time  situational  assessment  and  to  the  execution  of  the 
mission.  It  bears  no  relationship  to  the  maturity  of 
technology  being  used  in  the  ACTD. 

Critical  Military,,  feed 

Again,  the  criterion  used  by  the  IG  to  judge  military  need 
was  not  valid.  The  military  importance  of  this  capability  to 
the  warfighter  was  identified  by  the  Army,  the  Navy,  and  the 
Marine  Corps.  The  critical  need  is  also  identified  in  the 
USACOM  Integrated  Priority  List  and  it  was  confirmed  by  the 
VCJCS  prior  to  approval  of  this  ACTD. 

4.  Page  29,  concerning  Kinetic  Energy  Boost  Phase  Intercept. 

Ccnmnentj. 

Technology  Maturity 

The  K£  BPI  (Phase  I)  ACTD  was  a  12  month  effort  to  develop 
the  CONOPs  and  assess  the  affordability  (cost  effectiveness) 
of  the  proposed  concept.  The  technology  of  the  KE  BPI  (Phase 
I)  ACTD  was  not  an  issue  because  this  effort  was  accomplished 
by  performing  modifications  of  existing  man- in- the- loop  and 
digital  simulations.  The  concept  was  determined  to  be 
technically  achievable,  but  operationally  unaffordable.  The 
KE  BPI  (Phase  II)  ACTD  was  not  approved.  The  report  should 
be  revised  to  reflect  that  the  approved  KE  BPI  ACTD  was  for 
Phase  I  only. 

Critical  Military  ..Hesd 
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The  need  for  a  capability  to  engage  tactical  ballistic 
missiles  during  boost  phase  was  confirmed  in  writing  by  the 
Director,  Operational  Plans  and  Interoperability  (J-7),  Joint 
Staff  prior  to  the  approval  of  this  ACTD. 

Timeframe 

The  initial  concept  for  this  ACTD  required  a  total  of  four 
years  (FY95-98)  .  Loss  of  funding  stretched  the  program  into 
FY99.  However,  the  approved  ACTD  was  for  Phase  I  only,  which 
required  slightly  less  them  one  year,  not  almost  six  years  as 
indicated  in  the  draft  audit  report.  While  this  was  less 
than  the  typical  timeframe,  it  was  consistent  with  the 
objective  of  this  ACTD  and  was  therefore  appropriate. 


5.  Page  29-30,  concerning  Low  Life-Cycle  Cost  Helicopter. 

Comment! 

Critical  Military  Need 

The  serious  need  for  this  capability  was  originally 
identified  by  two  CINCs.  The  CNO  confirmed  the  need  in  two 
memoranda  to  the  DUSD(AT)  .  The  Vice  Chairman  also  strongly 
supported  the  need  during  a  meeting  with  the  DUSD  (AT)  . 

Advancing  Technology 

The  audit  report  questioned  this  ACTD  because  it  was  not 
"advancing  technology  to  meet  a  military  need,  *  but  rather 
was  to  demonstrate  an  acquisition  strategy,  the  viability  of 
commercial  helicopters  to  meet  the  Navy  requirements.  This 
comment  does  not  relate  to  any  of  the  selection  criteria. 
There  is  a  critical  need  and  it  is  being  addressed  with 
technology  that  is  sufficiently  mature.  In  ACTDs  and 
throughout  defense  acquisition,  there  is  strong  emphasis  on 
the  consideration  of  commercial  items  prior  to  the 
commencement  of  development  effort.  The  issue  is  the 
military  utility  (e.g.  effectiveness  and  suitability)  of  the 
commercial  item.  An  ACTD  is  well  suited  to  making  that 
determination  of  military  utility.  The  intent  of  the  ACTD 
process  is  well  served  when  a  commercial  item  is  evaluated  in 
a  military  application  and  proven  to  have  high  utility.  The 
report  should  be  revised  to  remove  this  ACTD  from  the 
questionable  category. 

6.  Page  30,  concerning  the  Rapid  Force  Projection 
Initiative. 

Comment.;. 

Technology  Maturity 
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The  models  and  simulation  used  in  this  ACTD  are  vital  to  the 
objectives  of  the  ACTD.  They  are  being  used  to  plan  the 
exercises,  develop  CONOPs,  assess  utility,  and  explore  paths 
for  evolution  of  the  ACTD  capability.  They  are  not  being 
used  to  compensate  for  core  capabilities  that  are  immature. 

Computers  -  All  computer  hardware/software  that  hosts  the 

Command  and  Control  <C2)  are  either  Army  Common 

Hardware /Software  (CHS)  or  Commerc ial -Of f -The- Shelf  (COTS) . 

C2  -  The  C2  systems  and  the  local  area  network  that  will  link 
these  systems  are  either  already  fielded  or  will  be  fielded 
to  experimental  units  before  the  RFPI  ACTD  field  experiments . 
The  Applique  C2  capabilities  that  will  be  used  for  RFPI 
consist  of;  (a)  Task  Force  XXI  capabilities  that  will  be 
demonstrated  in  that  Advanced  War  fighting  Experiment  more 
them  a  year  before  the  RFPI  experiments;  and  (b)  enhancements 
to  the  software  that  are  required  to  address  RFPI  unique  C2 
requirements.  The  enhancements  require  software  development, 
but  that  development  will  follow  current  Army  architecture 
and  software  standards  and  will  require  no  software 
technology  development. 

Communications  -  The  backbone  of  the  digital  communications 
system  will  be  the  Very  High  Speed  Integrated  Circuit  (VHSIC) 
Enhanced  Position  Location  Reporting  System  ( EPLRS )  which  is 
an  upgrade  to  the  currently  fielded  EPLRS.  The  VHSIC  EPLRS 
is  scheduled  to  be  fielded  by  the  Army  starting  in  mid  FY97. 

A  "tactical  internet*  architecture  is  to  be  used  to  link  the 
VHSIC  EPLRS  units.  This  architecture  is  based  on  proven 
commercial  LAN  algorithms  and  routing  schemes.  It  will  be 
used  in  the  TFXXI  AWE  prior  to  the  RFPI  experiments . 
Furthermore,  a  backup  communications  architecture  is  also 
being  developed  in  case  the  tactical  internet  architecture  is 
not  satisfactory.  The  backup  architecture  is  standard 
communications  architecture  and  is  based  on  the  inherent 
capabilities  of  the  VHSIC  EPLRS. 

While  the  C4I  capability  required  for  RFPI  is  complex  and 
requires  some  development  activity,  the  effort  within  this 
ACTD  is  integration  effort,  not  technology  development.  The 
technologies  being  used  will  be  proven  prior  to  their  use  in 
RFPI.  The  risks  for  this  ACTD  have  been  defined  and  risk 
management  measures  are  in  place.  Furthermore,  the  risk 
levels  are  considered  to  be  reasonable  and  have  been 
accepted.  The  report  should  be  revised  to  remove  this  basis 
for  assessing  technology  maturity. 

Critical  Military  Need 

The  critical  need  to  enhance  the  survivability  and  lethality 
of  our  light  early  entry  forces  when  faced  with  an  armor 
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threat  was  identified  by  the  Army  in  1994  and  the  need  was 
then  confirmed  by  the  VCJCS  prior  to  approval  of  this  ACTD. 

7.  Page  31,  concerning  Counter  Proliferation. 

Comment ? 

Critical  Military  Need 

The  priority  need  for  the  capability  to  hold  at  risk 
facilities  designed  to  process  and  store  weapons  of  mass 
destruction  has  been  identified  in  the  Deputy  Secretary  of 
Defense  Report  to  Congress  on  Counter  Proliferation.  It  has 
also  been  stated  by  CINCEUR,  the  user  for  this  ACTD,  and 
confirmed  by  the  VCJCS. 

8.  Page  31,  concerning  Cruise  Missile  Defense  (Phase  I). 

Comment 1 

Critical  Military  Nasd 

The  Navy  initially  highlighted  the  critical  need  to  extend 
the  capability  of  ships  to  defend  against  low  flying  cruise 
missiles  and  this  need  was  supported  by  CINCPAC.  This  need 
was  confirmed  by  the  VCJCS  prior  to  the  approval  of  this 
ACTD. 

9.  Page  31,  concerning  the  Precision/Rapid  Counter  Multiple 
Rocket  Launcher. 

Comment : 

Critical  MiUtarvJEfeefl 

The  critical  need  for  a  capability  to  counter  the  North 
Korean  240  mm  multiple  rocket  launcher  threat  was  initially 
identified  by  CINC  USFK  in  discussions  with  the  Deputy 
Assistant  Secretary  of  the  Army  for  Research  &  Technology. 
This  need  was  confirmed  by  the  VCJCS  prior  to  the  approval  of 
the  ACTD. 
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THE  VICE  CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 

WASHINGTON.  O.C.  20318-0001 


CM— 1595— 97 
28  February  1997 


MEMORANDUM  FOR  THE  INSPECTOR  GENERAL,  DEPARTMENT  OF 
DEFENSE 

Subject:  Audit  Report  On  Advanced  Concept  Technology  Demonstrations 
(Project  6AB-0015) 


1.  Thank  you  for  the  opportunity  to  review  the  findings  of  the  draft  report. 
Advanced  Concept  Technology  Demonstrations  (ACTDs)  are  a  key  element  in 
the  DOD  Science  and  Technology  (S&TJ  and  acquisition  processes  and  any 
attempts  to  improve  their  ability  to  fulfill  warfighter's  requirements  are 
encouraged.  Although  the  report  asserts  ambiguity  in  definitions  for  “mature 
technology*  and  “critical  military  need,-  the  Joint  Staff,  JROC,  and  OSD 
Advanced  Technology  staffs  have  worked  in  concert  to  understand  the  technical 
risks  and  warflghting  value  of  each  program.  The  JROC  now  prioritizes  all 
ACTD  candidate  programs.  While  we  find  fault  with  some  initial  assumptions 
and  some  specific  observations  of  the  investigation  (i.e..  the  JROC  did  in  fact 
assess  and  validate  the  military  need  of  the  ongoing  ACTDs  with  the  JWCA 
process),  most  of  the  final  recommendations  pass  the  “reasonable  man*  test 
and  are  endorsed.  The  ACTD  process  is  still  young  and  improvements  to  the 
ACTD  Master  Plan  are  to  be  expected  and  encouraged  as  it  evolves  into  a  more 
efficient  process. 

2  Regarding  the  report’s  major  recommendations,  a  summary  of  Joint  Staff 
positions  is  enclosed.  The  Joint  Staff  supports  the  1G  recommendation  to 
establish  an  ACTD  Working  Group  or  Integrated  Concept  Team  which  will 
identify  process  improvements  to  speed  advanced  technology  to  the  warfighter. 


General,  USAF 
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